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Executive Summary 


The overall purpose of the research studies described in this report was to investigate the 
quality of the administration of the Partnership for Assessment of Readiness for College and 
Careers (PARCC) field test during the spring of 2014. These research studies were conducted for 
the purpose of formative evaluation. Findings from these studies are being used to identify 
threats to the validity of PARCC test administration so that those threats can be addressed and 
reconciled prior to operational test administration. Throughout this report threats are 
identified, suggestions for ameliorating those threats are provided, and actions that have been 
implemented, or are being implemented, by PARCC to address those threats are provided. The 
actions that PARCC is taking to address the threats come from the PARCC Field Test: Lessons 
Learned report (PARCC, 2014).* 


The usefulness and interpretability of test scores require that tests be administered according 
to established procedures; without such standardization of test administration, the accuracy 
and comparability of score interpretation is impaired, threatening the validity of interpretations 
of these scores for their intended uses (AERA, APA, NCME, 2014, p. 111). Consequently, quality 
test administration is integral to the Theory of Action (TOA) for PARCC. A TOA indicates the 
intended uses and expected impact of an assessment system, and informs testable claims 
related to the interpretation of test scores (i.e., the Interpretive Argument). The evidence to 
support those claims and assumptions represents the Validity Argument. The findings included 
in this report speak to the validity argument for the Administration phase of the PARCC TOA. 
Test Administration is the fourth “phase” in the larger PARCC TOA. The specific claims from the 
Administration phase of the PARCC TOA are presented in Figure 1. 


Several sources of information were used in evaluating the claims from the Test Administration 
phase in the PARCC TOA. First, we observed a sample of test administration sessions and 
completed an observational checklist and debriefing interview with individuals involved in test 
administrations at the observed schools. The observation checklist and debriefing interview 
protocol were created from HumRRO’s review of the administration training materials and 
administration manuals. Second, we analyzed responses to Test Administrator (TA) Surveys. 
Finally, we analyzed responses to surveys completed by students who participated in the field 
test. Copies of the observational checklist, debriefing interview protocol, and survey 
instruments are provided in appendices to this report. The conclusions that follow were drawn 
from analyses of these specific data sources. 


. http://www.parcconline.org/parcc-states-release-lessons-learned-report 
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4. Administration 


* Claim 1: Administrators 
and teachers* are prepared 
to administer the 
assessments as intended. 


* Claim 2: Teachers* have 
the resources and supports 
to administer the 
assessments. 


* Claim 3: Technology 
improves and facilitates the 
assessment experience. 


* Claim 4: Students respond 
to items as intended. 


Quality of Test Administration 


Figure 1. Claims from the Administration Phase of the PARCC Theory of Action. 
The Quality of Test Administration research studies focus specifically on Test Administrators (TAs). Consequently, 


for the purposes of this report, Claims 1 and 2 are investigated specifically for TAs. 


The findings from the research studies on test administration provide evidence in relation to 


the claims from the Test Administration phase of the PARCC TOA. A summary of these findings 


is presented next. Throughout this report, references are made to activities that address the 


Standards for Educational and Psychological Testing (hereafter referred to as the Standards 


[AERA, APA, & NCME, 2014]). 


Claim 1: TAs are Prepared to Administer the Assessments as Intended 


In order for this claim to be true important assumptions that must be verified are: (a) TAs are 


trained to administer the assessments and (b) training is effective. Findings related to several of 


the research questions provide evidence for these assumptions and for the overall claim of 


preparedness. 
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The research questions that provide evidence related to Claim 1 are: 


Research Question 2: To what degree do the TAs follow the protocols and instructions? 
(addresses Standards 6.1 and 6.2) 
Research Question 6: Are there any apparent attempts to record or copy test materials 
including the test questions by students or others? (addresses Standards 6.6 and 6.7) 
Research Question 8: If any disruptions, interruptions, or other problems occurred, did 
the test administrators deal with the issue appropriately and effectively? (addresses 
Standards 6.1 and 6.3) 

o Research Question 8a: Were the TAs’ actions informed by training, by background 

experience, or a combination of the two? (addresses Standard 6.1) 

o Research Question 8b: Would a different approach have been more effective? 
Research Question 9: Was security of test materials maintained at all times? (addresses 
Standards 6.6 and 6.7) 


The findings from the research studies on test administration provide the following 


evidence in relation to the research questions mapped on to Claim 1: 


The overall participation rates in the PARCC trainings indicate that the majority of Test 
Administrators (TAs) participated in some form of training; however, findings suggest 
that some TAs might not have received training. 

Of those indicating completion of online training, 78% completed the “Computer-based 
Testing Test Administration” module (i.e., 58% of all TAs who responded to the survey) 
and 41% completed the “Accessibility Features and Accommodations with Computer- 
based Testing” module (i.e., 30% of all TAs who responded to the survey). These were 
the two most frequently completed online modules. 

Less than half of the TAs had ever administered a computer-based test at the district or 
state level. Therefore, for the majority of TAs this suggests that their knowledge of 
computer-based test administration likely came from one or more of the trainings for 
PARCC. Although TAs were not asked to indicate whether they had ever taken a 
computer-based test in a proctored setting, it is possible that TAs might have assumed 
that they already knew enough about administering computer-based tests from prior 
experience taking computer-based tests in proctored settings. 

Of those who participated in online training, three-quarters agreed that it effectively 
informed them of the protocol to follow when administering assessments, and slightly 
more than half agreed that the online training prepared them to resolve basic problems 
related to technology (e.g., logging students in, exiting the test, etc.). 

Most TAs followed the major protocols and instructions in the TA Manual during test 
administration. However, there were a few minor exceptions. First, in nearly a fourth of 
the observed sessions the TAs did not post the “Testing — Do Not Disturb” sign. Second, 
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in over a third of the sessions the observers noted that TAs had not arranged computers 
so that there were partitions and/or sufficient space between computers (this has 
implications for test security.) Third, in approximately half of the observed sessions TAs 
did not keep track of time using a timing box’ as instructed in the TA Manual. Finally, the 
most notable deviation from test administration instructions was that in nearly half of 
the observed sessions the TAs did not read the script verbatim. Observer notes indicate 
that the most common modifications to the script were: (a) skipping repetitive parts, (b) 
changing words (e.g., changing “session” to “part”), and providing additional guidance 
beyond what was in the script (e.g., for logging into TestNav, for using headphones). It 
should be noted that TAs were allowed to clarify the script after reading the entire 
script. In general, these deviations are considered minor. 

e Overall, TAs handled the technology-related problems to the best of their ability. For 
example, they consulted their TA Manual for assistance, they worked with their 
Technology Coordinators, and they contacted the Help Desk for assistance. Nonetheless, 
in some instances the technological issues were substantive, and occasionally the issues 
were not resolved despite the TAs’ best efforts. It should be noted that there were no 
system-wide technology issues during the field test. System-wide technology issues may 
stem from platform or infrastructure weaknesses, and may include overall performance 
degradation (e.g., long wait time between clicking next and next question appearing on 
the screen) or prolonged testing interruptions.* During the field test, there were no 
technology issues that impacted the platform for all. Most technology issues that did 
occur were local (e.g., firewall settings needed to be changed, computer settings 
needed to be adjusted, or students needed help logging in) — an expected result when 
school districts introduce computer testing for the first time, as was the case in most 
PARCC states. The majority of issues were quickly and easily resolved. 


Recommendations that stem from the above findings related to Claim 1 are: 


e Increase participation in training, including training on accessibility features and 
accommodations. 


* The TA Manual instructs TAs to, “Draw a timing box on the board.” The manual includes an example timing box 
for ELA/Literacy and Math. The examples differ slightly for ELA/Literacy and Math, but both boxes include lines on 
which TAs are to enter: Session Name, Session Time, Seal Code, Starting Time, Stopping Time, and Time 
Remaining. 

? It should be noted that throughout this report there is evidence reported from the Test Administration studies of 
devices working slowly. However, “the referenced intermittent slowness is not evidence of a system-wide issue 
that impacts all users. Local bandwidth and infrastructure constraints play a key role in the delivery of online 
assessments. Thus, there are instances where local capacity can be maxed out on local network activity that is 
present on the network from typical instruction and administration activities, plus the bandwidth required for 
taking the PARCC assessment. To avoid local constraints, the bandwidth requirements outlined in the PARCC 
Technology Guidelines should be followed.” Email communication from Pearson technology contact on January 26, 
2015. 
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e Provide more in-person training. 

e Condense and synthesize trainings and training material. 

e Provide clearer training on managing testing time, including managing additional time 
allowed, to ensure that all students have the appropriate allotted time for completing 
the test. 

e Implement efforts to ensure that TAs are consistent across sites in reading the script in 
the TA Manual (see Claim 2 for recommendations for improving the TA script). 


Recommendations specific to Computer/tablet-based test (CBT) administration: 


e Provide more (clearer) training on: (a) procedures for logging on and off, (b) disabling 
pop-up blockers, (c) use of headphones, and (d) use of tools. 

e Provide additional practice with a “mock” test for both students and TAs, which would 
include practice accessing and navigating the TestNav system. 

e Provide TAs with additional support from technology staff to help prepare for and deal 
with the technology issues. 


Actions by PARCC to Address Concerns Related to Claim 1 


In light of the information gathered during field testing, PARCC is already taking several actions 
to address concerns and potential threats to the claim that “TAs are prepared to administer the 
assessments as intended.” Actions that are being implemented by PARCC to improve the 
preparedness of TAs include: 


e Providing TA training modules earlier in the school year to facilitate advance 
preparation. 

e Splitting modules into task based roles so that Test Administrators (TAs) and Test 
Coordinators (TCs) can clearly and easily identify which tasks they are responsible for; 
manuals include separate symbols for TA tasks and TC tasks for clarity. * 

e Splitting up training modules into chapters by topic with search fields to allow easier 
navigation and to facilitate locating information. Providing a summary of where to find 
key training information. 

e Enhancing the guidance and documentation around the SystemCheck tool to ensure 
wider use and understanding. 

e Revising tutorials to include full array of tools, accessibility features, and item-computer 
interactions. 


“The PARCC states have made several improvements to available modules based on field test feedback, including: 
Student Registration Import (September 19, 2014); PearsonAccess"™ (October 9, 2014); Paper-based Testing for 
Test Coordinators (October 10, 2014); Paper-based Testing for Test Administrations (October 13, 2014); 
Introductions to Training (October 20, 2014); Accessibility Features and Accommodations (October 21, 2014). 
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e Updating the Accessibility Features and Accommodations training module, and updating 
the PARCC Accessibility Features and Accommodations Manual. 

e Expanding practice tests to include paper and to include both the Performance-based 
Assessment (PBA) and the End-of-Year (EOY) Assessment in both content areas 
(Mathematics and English-Language Arts) so that TAs can become more familiar with 
the tests prior to administration. 


These actions help to ameliorate concerns identified during the field test, and will help to 
ensure that TAs are better prepared to administer the PARCC assessments as intended during 
operational testing. 


Claim 2: TAs Have the Resources and Supports to Administer the Assessments 


For Claim 2 to be true important assumptions that must be met are: (a) resources and supports 
must be clear, sufficiently detailed, and easy to follow, and (b) resources and supports must not 
be overly burdensome for TAs to use and apply. 


The research questions that provide evidence for this claim and these assumptions are: 


e Research Question 1: To what degree do the test administrators (TAs) find the 
instructions clear, sufficiently detailed, and easy to follow? (addresses Standard 4.15) 

e Research Question 10: Did the test administration create minimal disruption to the 
school and staff? 


The findings from the research studies on test administration provide the following evidence in 
relation to the research questions mapped on to Claim 2: 


e Approximately two-thirds of TAs agreed that the policies and procedures in the TA 
Manual were easy for them to understand. 

e Slightly fewer (55% - 63%) agreed that the instructions (including the scripts for 
administering the assessments) were easy to implement. 

e Feedback provided by the TAs indicates that there were some inconsistencies between 
what appeared in the TAs’ scripts and what appeared on students’ screens or booklets. 
Such disconnects contributed to some confusion in navigating through the tests. 

e Some TAs also indicated that the instructions for submitting answers and completing 
tests (CBTs) were not easy to understand. 

e Some TAs indicated that the TA Manual was confusing at times because large parts of 
the manual were not relevant to the particular grade being tested. This required the TAs 
to skip back and forth between sections of the lengthy manual to find the sections that 
were relevant to the grade being tested; TAs reported that having to skip around made 
it easy to get lost in the script. 
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e Some TAs reported that the instructions for timing the test, including the instructions 
for additional time, were not as clear as they would have liked. 

e Some TAs reported confusion over ambiguous terminology in the TA Manual, such as 
the terms “sections” and “sessions,” which were used interchangeably and 
inconsistently. 

e Some TAs reported that the “volume of materials” (e.g., online training, manuals, and 
various communications and updates), and the time required to synthesize and digest 
all that information was overly burdensome. 

e Some TAs reported confusion about the read aloud accessibility 
feature/accommodation (text-to-speech vs. human reader) described in the Accessibility 
Features and Accommodations Manual. 

e Points of confusion that were specific to the paper-based test (PBT) administration 
were: (a) no page numbers in the student instruction books (students did not know 
which page to turn to), and (b) confusion about the instructions for when calculator use 
was allowed and not allowed. 


Recommendations that stem from the above findings related to Claim 2 are: 


e Check instructions in the TA script against instructions on students’ screens/booklets to 
ensure there are no inconsistencies between what appears in the script and what 
appears on students’ screens/booklets. 

e Clarify instructions in the TA manual for (a) managing time, including additional time, 
and (b) submitting answers and completing tests (CBTs). 

e Remove and/or reconcile ambiguous terminology in the script (e.g., “sessions” and 
“sections”). 

e Consider making calculators available for entire sections as opposed to parts of sections 
so that it will be easier for TAs to monitor their use (PBT version). 

e Condense and synthesize the training and the training materials into a more digestible 
amount and format. 

e Create grade-specific scripts. 

e Develop a summary or checklist of major steps and key information for TAs. 

e Clarify instructions in the Accessibility Features and Accommodations Manual on the 
read aloud accessibility feature/accommodation. 


Actions by PARCC to Address Concerns Related to Claim 2 


In light of the information gathered from field testing, PARCC is already taking several actions to 
address concerns and potential threats to the claim that “TAs have the resources and supports 
to administer the assessments.” The “resources” and “supports” (beyond the trainings 
discussed under Claim 1) that were the focus of these research studies were the TA Manual 
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protocols and script. Actions that are being implemented by PARCC to improve the TA Manual 
are: 


e Shortening the TA Manual by reducing redundancies so that there is not as much 
information to sort through. 

e Working with grade level experts to ensure administration scripts are grade appropriate, 
as scripts are being split into grade bands within the TA manual to facilitate their use. 

e Updating the TA manual to include simple and direct checklists of tasks to complete 
before, during, and after testing. 

e Updating the TA manual to include more explicit directions, including graphics and icons 
to increase user understanding. 

e Changing the terminology for testing sessions to clarify terms for testing. 

e Clarifying instructions for calculator use to make it easier for TAs to monitor calculator 
use; for CBTs, changes are being made to the computer-based interface to make it 
easier to see if the student is in a calculator section; for PBT, a calculator icon will be 
added to the pages in the test booklet to make it easier for TAs to monitor. 

e Updating instructions in the TA manual so there are no disconnects in directions for the 
“Submit” function and the “Exit & Save” function. 

e Simplifying the testing time policies described in the TA manual so that it is easier for 
TAs to understand how to manage testing time. 

e Sending the TA manual to PARCC and state representatives for review to ensure that 
procedures and directions are simple and clear. 

e Updating the Accessibility Features and Accommodations Manual. 


The above actions, coupled with actions to streamline and simplify TA training, help to 
ameliorate concerns identified during field testing with the PARCC resources and supports. 


Claim 3: Technology Improves and Facilitates the Assessment Experience 


For this claim to be true an important assumption that must be met is that the technology must 
work as intended. If the technology does not work as intended—for example, if there are 
functionality problems and/or delays in its application, then the veracity of this claim is 
threatened. 


The evidence for this claim comes directly from Research Question 7a: 


e Are there any hardware or software related problems during the test administration? 
(addresses Standards 6.3 and 6.4) 


The findings from the research studies on test administration provide the following evidence in 
relation to the research question mapped on to Claim 3: 
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e Nearly 90% of TAs reported that they encountered one or more problems related to 
technology during the assessment administration. 

e TAs reported that they most frequently encountered problems with logging into the 
assessment. The next most frequently encountered problems reported by TAs were 
with devices that stopped working (“shutting down”), devices working slowly, and losing 
internet connection, respectively. Loss of student data occurred in many instances when 
devices locked up and users were kicked out of the system. 

e Overall, approximately half of the students indicated that they encountered technology- 
related problems during their test. 

e Students reported higher occurrences of devices working slowly than of devices 
“stopping” (i.e., shutting down) entirely. 

e Some students also reported difficulties with dragging or moving objects on the screen 
and difficulties making changes to answers. Difficulty making changes to answers was 
reported at a higher frequency for the Mathematics tests than for the English Language 
Arts (ELA) tests. Some students also reported difficulty entering math symbols and 
numbers into the equation editor on the mathematics tests. 


There was a reduction in the pervasiveness of the technological problems from the time 
in which the initial PBAs were administered to the subsequent administration of the EOY 
assessments. This suggests that improvements were made overtime, and that problems 
can be addressed and reconciled for future administrations. Although it should be noted 
that the PBAs included more technologically enhanced items than the EOY assessments. 


Of those students who reported using the highlighter tool and the magnifying tool, most 
indicated that they were easy to use. 


Recommendations for TAs that stem from the findings for Claim 3 are: 


e Prepare ahead of time to help discover and address potential technology difficulties 
prior to test administration. 

e Utilize the support available from district technology staff. 

e Investigate the PARCC SystemCheck tool to ensure that it is catching technological issues 
prior to test administration. 


An additional recommendation is to: 


e Update the training provided to TAs on handling technology-related issues to address 
the common technological issues encountered during field testing (e.g., logging on, 
submitting tests, disabling pop-up blockers, etc.). 
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Actions by PARCC to Address Concerns Related to Claim 3 


In light of the information gathered from field testing, PARCC is already taking several actions to 


address concerns and potential threats to the claim that “Technology improves and facilitates 


the assessment experience.” Although local districts are responsible for their own technology 


(e.g., devices, internet connectivity, etc.), the PARCC consortium is working to provide districts 


and schools with additional supports to improve the local technology experience. Actions that 


are being implemented by PARCC to address technology-related issues include: 


Improving exiting procedures in the test delivery system so that students are not locked 
out of tests prematurely. 

Conducting an extensive verification and validation process of the delivery system prior 
to the operational assessments to confirm that the system can support the thousands of 
simultaneous test-takers without interruptions and without loss of student answers. 
Working with a vendor to conduct a large-scale “dress rehearsal” to identify issues with 
the highlighter tool and other tools (e.g., magnifier and drag-and-drop) and develop 
solutions to further improve functionality before operational testing. 

Reviewing and improving information shared with school districts and schools about the 
technology requirements and about how to set up devices to reduce blockage of videos 
due to filter settings on testing devices. 

Providing more time for the infrastructure trial by making it available to schools earlier. 
Enhancing guidance and documentation around the SystemCheck tool to ensure wider 
use and understanding of the tool. The SystemCheck tool allows users to validate that 
their testing workstations meet the minimum requirements needed to run TestNav8 for 
the assessments and evaluate bandwidth capacity for internet and proctor-caching 
connections. 

Updating the training modules available to guide users in conducting an infrastructure 
trial and proctor caching to ensure that the modules are more comprehensive and 
provide clearer step-by-step instructions. 

Providing a new technology preparedness training module to help organizations 
understand, manage, and make decisions in preparing school technology to be used for 
online testing. 


Launching an updated interface for PearsonAccess (PearsonAccess”™ 


) to make it more 
user-friendly. 

Improving the equation editor in the Mathematics assessments so that it is easier for 
students to use. 

Providing practice tests to allow students and teachers to become more familiar with 


the equation editor and other tools. 
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e Increasing the knowledge of Call Center staff so that they are better equipped to help 
TAs resolve technology-related problems. 

e Increasing the number of representatives staffed at the Call Center. 

e Creating a “Quick Start Checklist” to have a cohesive step-by-step guide to start up. 

e Adding a live chat feature to the Customer Support Center to allow additional avenue 
for responses. 

e Linking the call-in process to PearsonAccess”™ user profiles to limit time spent 
identifying callers. 

e Bringing in an independent vendor to test the platform and identify performance issues 
needing improvement. 

e Conducting “load testing” and other tests to pinpoint performance issues and solutions. 

e Improving information to the field on minimum technology specifications and how to 
set up devices. 

e Developing clear, consistent, and comprehensive communication to be sent to states 
and districts regarding technology preparedness tools, resources, timelines, and 
support. 


These actions help to ameliorate concerns identified during the field test, and will help to 
ensure that technology improves and facilitates the assessment experience. 


Claim 4: Students Respond to Items as Intended 


Finally, for Claim 4 to be true, several assumptions must be verified. First, for students to 
respond to items as intended, they must understand the directions read by the test 
administrator (i.e., the script). Second, students must also understand the directions they read 
for the questions on the test. If the students do not understand the directions read by the TA 
and/or if they are confused by the directions they read on the test, then they may not respond 
to the items as intended. Third, to respond to items as intended, students need to have some 
engagement with the assessment. If they are not engaged or if they have a low level of 
engagement, then they might not sufficiently attend to items so as to respond to them as 
intended. Several factors might negatively impact student engagement, such as distractions 
caused by: (a) unauthorized visitors, (b) disruptive student behavior, and (c) technological 
problems, to name a few. 


The research questions that provide evidence for this claim and these assumptions are: 


e Research Question 3: Do the students appear to understand the instructions provided to 
them? (addresses Standards 4.16 and 6.5) 

e Research Question 4: To what degree are students engaged in taking the test? 
(addresses Standards 4.16, 6.3, 6.4, and 6.5) 
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Research Question 5: Is there any disruptive student behavior during the session? 
(addresses Standard 6.4) 
Research Question 7: Are there any interruptions during the session? (addresses 
Standards 6.3 and 6.4) 
o Research Question 7a: Are there any hardware or software related problems 
during the test administration? 
o Research Question 7b: What type of questions, if any, do students ask during the 
test administration? (also relevant to Standards 4.16 and 6.5) 


Overall, the findings from the research studies on test administration provide the following 


evidence in relation to the research questions mapped on to Claim 4: 


The majority of students (78% or more across the test types and administration modes) 
reported that they understood the directions (i.e., script) read by their TA. 

However, there was also converging evidence from the TA Surveys and from the 
observation study that the script read by the TAs was too lengthy, and that, as a result, 
some students had trouble staying engaged. 

Findings indicate that the majority of students had some difficulty in understanding at 
least some of the directions they read for the questions on the test. This was particularly 
true for the Mathematics tests with only about 15% of students indicating that it was 
hard to understand the directions for the questions on the test “almost never.” For ELA 
tests, about a third of the students indicated that it was hard to understand the 
directions “almost never.” 

Findings from the research studies indicate that the TAs effectively kept distractions 
such as, outside visitors and disruptive student behaviors, to a minimum. 

Some students reported that they “liked” the more interactive nature of the CTBs. 

The biggest distraction, and thus impediment to student engagement, appears to have 
been the technology-related problems, which sometimes caused delays in testing 
and/or resulted in the loss of student data. As a result, some students reported that 
they skipped items that were not functioning correctly or that they rushed to finish the 
test because of the delays caused by the technology problems. 


Recommendations that stem from the above findings related to Claim 4 are: 


Shorten the scripts by removing repetitive content, particularly when TAs are 
administering back-to-back sessions. 

Provide students and TAs with more exposure to practice items so that they have 
greater familiarity and understanding of what the instructions for the questions on the 
test are asking them to do. Many of the test items are novel item types, which makes 
practice opportunities particularly helpful. 
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e Allow students to read a book as soon as they have completed their test so that they are 
less likely to distract other students who are still testing. 

e Take steps to address technology-related problems, and thereby improve student 
engagement with assessment. 

e Revising general directions on the tests, especially the mathematics tests, to make them 
clearer. 


Actions by PARCC to Address Concerns Related to Claim 4 


In light of the information gathered from field testing, PARCC is already taking several actions to 
address concerns and potential threats to the claim that “Students respond to items as 
intended.” Many of the actions already identified under Claims 1, 2, and 3 will help to ensure 
that students respond to items as intended. Additional actions not already addressed above 
that PARCC is taking to help ensure that students respond to items as intended are: 


e Expanding tutorials to include the full array of tools, accessibility features, and item- 
computer interactions.” 

e Revising tutorials to make them more grade-appropriate. 

e Expanding practice tests to include paper and to include both components (PBA and EOY 
in both content areas) so that students can become more familiar with the tests prior to 
administration. 

e Updating the test scripts in the manual to contain more grade-appropriate language, 
which should help to improve students’ attention to the scripts and understanding of 
the information conveyed in the scripts. 


These actions combined with the actions described under Claims 1, 2 and 3 will help to address 
threats to the claim that students respond to items as intended. 


Conclusion 


These findings do not provide an exhaustive investigation of the validity evidence for the PARCC 
test administration. Consequently, the evidence presented here should not be taken as the final 
word on the PARCC field test administration. Nonetheless, the findings presented here do 
represent the foundation of the validity argument for the PARCC field test administration. 
There were some positive findings from this investigation; there is also some evidence 
indicating a lack of support for the claims in the Test Administration phase in the TOA. Because 
lack of support for earlier goals may undermine the validity of subsequent goals, the evidence 


° The planned release dates for the expanded student tutorials for CBT and PBT are: High school tutorials: 
November 2014 (available at parcc.pearson.com/tutorial); Grade 3 — 8 tutorials: January 2015 (available at 
parcc.pearson.com/tutorial). Online practice tests for the mathematics EOY assessment and the ELA PBA 
assessment are currently available. 
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reported here represents concerns for the validity of PARCC test score interpretations for 
future operational assessments for their intended uses if these issues persist in future 
administrations. These issues should be addressed, to the extent feasible, prior to operational 
testing. The actions already being implemented by the PARCC consortium will help to ensure 
that those threats are reconciled prior to operational testing. The planned Test Administration 
Study for operational testing (i.e., TA Survey, Student Survey and Site Observations) will help to 
verify whether these corrective actions have been effective. This validity argument should be 
updated as new evidence is collected. 
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Findings from the Quality of Test Administration Investigations: 
PARCC Field Test 


Purpose 


The purpose of the research studies described in this report was to investigate the quality of the 
administration of the Partnership for Assessment of Readiness for College and Careers (PARCC) 
field test in Mathematics and English Language Arts (ELA)/Literacy during the spring of 2014. 
These research studies were conducted for the purpose of formative evaluation. Findings from 
these studies are being used to identify potential threats to the validity of PARCC test 
administration so that those threats can be addressed and reconciled prior to operational test 
administration. Throughout this report threats are identified, suggestions for ameliorating 
those threats are provided, and actions that have been implemented, or are being 
implemented, by PARCC to address those threats are provided. All of the PARCC actions 
included in this report come from the PARCC Field Test: Lessons Learned report (PARCC, 2014).° 


Background 


Quality of test administration is integral to the Theory of Action (TOA) for PARCC. A TOA 
indicates the intended uses and expected impact of an assessment system and informs testable 
claims related to the interpretation of test scores (i.e., the Interpretive Argument). The evidence 
to support those claims and assumptions represents the Validity Argument. The findings 
included in this report speak to the validity argument for the Administration phase of the PARCC 
TOA. The Standards for Educational and Psychological Testing (hereafter referred to as the 
Standards) provide criteria for the development and evaluation of tests and testing practices as 
well as guidelines for assessing the validity of interpretations of test scores for the intended test 
uses (AERA, APA, & NCME, 2014, p. 1). Throughout this report references are made to findings 
that relate to the administration criteria from the Standards. 


The validity studies for the PARCC field test take an argument-based approach to validation, 
whereby the goal is to evaluate the extent to which the evidence supports or refutes the 
proposed interpretations and uses of test scores (Kane, 2006). This approach is consonant with 
the Standards on validity, which state that a rationale should be presented for the intended 
interpretation of test scores for a given use, and that that rationale should identify the 
claims/propositions that must be investigated to support the intended interpretation of test 
scores (AERA, APA, & NCME, 2014, p. 23). Per the Standards it is “incorrect” to use the 
unqualified phrase “the validity of the test.” Rather, validity is a matter of the validity of the 
interpretations and uses of test scores. Consequently, the evidence reported here represents 


: http://www.parcconline.org/parcc-states-release-lessons-learned-report 


Updated February 17, 2015 Page 1 


PARCC Quality of Test Administration 


concerns for the validity of PARCC test score interpretations from future operational 
assessments for their intended uses if these issues persist in future administration. 


PARCC scores are intended to be interpreted and used as an indication of whether students are 
on track in their learning to be successful in college and their careers.’ This is commonly referred 
to as “college and career readiness,” or CCR. Based on the stated claims and purposes of the 
PARCC assessments as described in PARCC’s publicly available documentation, a TOA for PARCC 
was developed with the end goal of college and career readiness (see PARCC Validity Studies 
Memorandum; Thacker, Sinclair, Wise, & Becker, 2014). The TOA for PARCC (see Figure 2) is 
organized as a series of interim goals that lead to an end goal. Each interim goal is a prerequisite 
for supporting subsequent interim goals. Again, the end goal of the PARCC assessment system is 
that PARCC scores in ELA and Mathematics provide an indication that students are, or are not, 
college and career ready (CCR), or on track to become CCR for younger students. In order to 
attain the end goal of CCR, each of the interim goals (denoted by the column headers in Figure 2) 
must be met. Lack of support for any of the interim goals potentially undermines the validity of 
the system to meet its end goal. 


The findings in this report address the interim goal of Test Administration from the PARCC TOA 
(see Figure 2). In order to have a quality test administration, certain claims must be met. Those 
claims are depicted by the bullet points under the Administration heading in the TOA. They are: 
(a) Administrators and teachers® are prepared to administer the assessments as intended, 

(b) Teachers have the resources and supports to administer the assessments, (c) Technology 
improves and facilitates the assessment experience, and (d) Students respond to items as 
intended. There are numerous assumptions that underlie each of these claims. For example, if 
students are to respond to items as intended, then the Test Administrators (TAs) must follow the 
appropriate protocols and instructions for administering the test (this is Standard 6.1 from the 
Standards). The claims related to the Administration phase of the TOA served as a guide for 
identifying assumptions that must be met for those claims to be true. Those assumptions relate 
to the research questions guiding the investigations of the PARCC field test administration. The 
research questions and the investigations designed to address those questions were proposed 
by HumRRO to PARCC and revised after multiple rounds of feedback from PARCC leadership. 


” From PARCC website: http://www.parcconline.org/about-parcc. 

® The PARCC field test was administered by “Test Administrators” (typically teachers). Given that PARCC refers to 
the individuals administering the tests as “Test Administrators” as opposed to “teachers,” the current report treats 
all references to “teachers” in the Administration phase of the TOA as “Test Administrators.” 
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1 
Design 


eConnects to CCSS 

¢Determines whether 
students are CCR or on track 
to be CCR 

¢Measures full range of 
student performance, 
including high- and low- 
achieving students 

Tests are faster and more 
efficient to administer 

e/nforms instruction, 
interventions, and 
professional development 

¢Provides data for 
accountability, including 
measures of growth 

eIncorporates innovative 
approaches, including 
technology, that are 
interactive and engaging 

¢Administered on range of 
devices 

¢Provides multiple measures 
of student achievement 

eIncludes range of item types 

¢Captures critical-thinking, 
problem-solving, and 
communication skills 


Provides timely results 

Accommodations are 
appropriate for SWD and 
ELL students 

¢Assessments are accessible 
to SWD 


2 
Development 


¢Subject matter, 
presentation, and 
language use is free 
of potential bias and 
is acceptable to 
students, parents, 
and other community 
members 


¢ Items are of 
sufficient quality and 
rigor 

¢Teachers provided 
with instructional 
materials, 
professional 
development and 
other supports to 
enable them to 
effectively instruct 
students on CCSS 
curriculum so that 
students can 
demonstrate their 
achievement on the 
assessments 


¢Schools have the 
resources and 
infrastructure to 
implement and 
administer the 
assessments as 
intended 


eEducators from across 
the country are 
trained by PARCC to 
become leaders and 
experts to share their 
knowledge and 
expertise within their 
community 


3 


Implementation 


¢Teachers 
effectively 
instruct students 
on CCSS-aligned 
curriculum 


¢Communication 
plan clearly 
conveys to 
stakeholders 
the policies and 
practices 
essential for 
effective 
implementation 
of assessment 
system 


¢Students 
understand the 
format of the 
assessments 
and how to use 
the technology 


eTest vendors, 
departments of 
education, 
school districts, 
and schools are 
coordinated in 
the assessment 
process 


4 


Administration 


Administrators 
and teachers are 
prepared to 
administer the 
assessments as 
intended 


¢Teachers have 
the resources 
and supports to 
administer the 
assessments 


¢Technology 
improves and 
facilitates the 
assessment 
experience 


¢Students 
respond to 
items as 
intended 


eTimely scoring 


Rubric is 
diagnostic 


eScores 
accurately and 
reliably reflect 
student 
achievement 
on the assessed 
content 


Growth 
inferences 
provide 
accurate 
information on 
changes in 
student 
performance. 


¢Scores from 
multiple PARCC 
assessments 
provide both 
unique and 
complementary 
information. 


Cut score is 
indicative of 
college- and 
career- 
readiness 


eInferences 
across forms 
and years are 
appropriately 
comparable. 


Figure 2. Seven sequential phases from the simplified TOA for the PARCC assessment system. 
Note. Bold font indicates claims related to test score interpretation and italics indicate claims regarding impact/consequences of test scores. 


Updated February 17, 2015 


Quality of Test Administration 


6 


Reporting 


eScore 
reports are 
clear and 
easily 
understood 
by 
stakeholders 


Results 
reported ina 
timely 
manner 


eResults 
reported in 
score reports 
are 
actionable 


7 
LUEY=) 


¢Students use results 
to determine if they 
are on track to 
graduate ready for 
college and careers, 
and to identify 
where gaps may 
exist 


eTeachers use results 
to help inform 
instruction, and 
provide supports and 
interventions to 
students with 
readiness gaps 


¢Parents have timely 
information about 
the progress of their 
children 


eStates compare their 
results with other 
states to make 
decisions about their 
relative performance 
and use that 
information to better 
plan for and develop 
future workforce 


eNation compares its 
performance to 
other countries to 
make decisions 
about relative 
performance and use 
that information to 
better plan for and 
develop future 
workforce 
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Those research questions and the Standards that 
they address are: 


1. To what degree do the test administrators 
(TAs) find the instructions clear, sufficiently 
detailed, and easy to follow? (addresses 
Standard 4.15) 

2. To what degree do the TAs follow the 
protocols and instructions? (addresses 
Standards 6.1 and 6.2) 

3. Do the students appear to understand the 
instructions provided to them? (addresses 
Standards 4.16 and 6.5) 

4. To what degree are students engaged in taking 
the test? (addresses Standards 4.16, 6.3, 6.4, 
and 6.5) 

5. Is there any disruptive student behavior 
during the session (e.g., refusal to take the 
test or leaving the room without permission 
during the testing session)? (addresses 
Standard 6.4) 

6. Are there any apparent attempts to record or 
copy test materials, including the test 
questions, by students or others? (addresses 
Standards 6.6 and 6.7) 

7. Are there any interruptions during the 
session? (addresses Standards 6.3 and 6.4) 
a) Are there any hardware or software 

related problems during the test 
administration? 

b) What type of questions, if any, do students 
ask during the test administration? (also 
addresses Standards 4.16 and 6.5) 

8. If any disruptions, interruptions, or other 
problems occurred, did the TAs deal with the 
issue appropriately and effectively? (addresses 
Standards 6.1 and 6.3) 

a) Were the TAs’ actions informed by 
training, by background experience, or a 
combination of the two? (addresses 
Standard 6.1) 

b) Would a different approach have been 
more effective? 

9. Was security of test materials maintained at 
all times? (addresses Standards 6.6 and 6.7) 

10. Did the test administration create minimal 
disruption to the school and staff? 
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Additional research questions of interest, although not directly linked to the TOA, include: 


11a. Did the observers’ responses to the Test Administration Observation Checklist (TA-OC) 
adequately and accurately capture relevant aspects of the assessment session? 


11b. Did the test administrators’ responses to the Test Administrator Surveys adequately 
and accurately capture relevant aspects of the assessment session? 


11c. Did the students’ responses to the Student Surveys adequately and accurately capture 
relevant aspects of the assessment session? 


While the above list of research questions address all of the standards related to test 
administration from the Joint Standards, the list should not be considered an exhaustive list of 
research questions on test administration. Rather, these research questions represent a 
combination of the topics of greatest interest at the time of PARCC field test administration and 
those that were most feasible on which to collect information. As stated in the PARCC validity 
studies memorandum (Thacker et al., 2014), the validity argument should be annually updated 
to document the continuing collection of validity evidence for the PARCC assessments. 
Furthermore, if support for a claim is weak or the evidence is inconclusive, then that aspect of 
the testing program must be improved or additional evidence gathered. As such, the 
information reported here should not be treated as the final word on the validity argument for 
the PARCC field test administration. Rather, the findings presented here provide initial evidence 
on the claims for an effective test administration, and provide guidance on actions to take to 
address threats to an effective test administration. 


The next section of the report outlines the method used to collect evidence for the research 
questions on test administration. 


Method 


An effective test administration requires high quality training of TAs. The training of TAs 
establishes the foundation for a well-prepared administration team that follows standardized 
procedures and adapts as necessary when things do not go as planned. Effective test 
administration also requires clear and consistent test administration instructions, including 
consideration of various testing conditions and unplanned occurrences. A sound infrastructure 
allows the training and the instructions to be applied as designed. These prerequisites must be 
in place in order for students to respond to the test items as intended. Three separate research 
investigations were designed to collect information on these requirements for meeting the 
interim goal of effective test administration. They are: 1) observation visits to test 
administration sites, 2) surveys of TAs, and 3) surveys of student test-takers. These three 
investigations provide three different perspectives on the test administration, and thereby help 
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to triangulate the findings related to the research questions. The procedures, instruments, and 
participants used for each of these studies are described next. 


Study 1: Test Administration Observations 


Procedure. HumRRO researchers visited administration sites to observe the 
administration of the PARCC Performance-Based Assessment (PBA) and End-of-Year (EOY) field 
test during the spring 2014 testing window. We based these observations on a successful 
protocol that HumRRO uses to ensure test administration quality for the National Assessment 
of Educational Progress (NAEP). That protocol is composed of detailed observation checklists 
that trained HumRRO staff use to document task completion during a sample of field test 
administrations. HumRRO revised the protocol based on the PARCC Test Administrator Manual 
and guidance from PARCC leadership. 


Prior to conducting observations of testing in schools for any project, HumRRO conducts 
internal observer training for selected staff. Although staff used for PARCC observations were 
experienced, it is essential to provide observers information tailored to the project at hand. 
HumRRO conducted the training with staff via teleconference, with interested PARCC staff in 
attendance. The training covered the following topics: 


e Anoverview of PARCC and the 2014 field test administration 

e PARCC assessment system and materials 

e What to expect when observing 

e Observation checklist (detailed review) 

e Observer conduct during school visit 

e Coordination with the sampled schools 

e What to take with you (e.g., observation forms, job aid, PARCC manual excerpts) 


A key aspect of the training was a detailed review of the observation checklist (described 
below) to ensure that observers developed a common understanding of each “task” (i.e., the 
activities performed by the TAs) and that the ratings were consistent across observers. Tasks 
were discussed to ensure observers understood what they would see when the task is 
conducted, or in some cases, how they would know if a task is completed. This is an important 
discussion since there are times a task can be ambiguous. For example, the task, “Monitor 
student progress using administration computer.” If the observer can see the TA looking at 
usage screens during the assessment, as opposed to responding to a student computer issue 
(i.e., resetting TestNav), then the observer was trained to rate the task as “Met” on the 
observation checklist. 


Additionally, procedures for interacting with the schools, school personnel, and students were 
reviewed. Observers were trained to not interfere with or provide guidance to test 
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administrators and not to interact with students, instead striving to be as unobtrusive as 
possible. 


The HumRRO observers assigned to visit schools were directed to contact the school’s PARCC 
test coordinator to verify the testing date and time, address any last minute logistical details, 
and again to provide assurance that we were not visiting to evaluate the school or their staff. 
Observers arrived at the schools wearing their PARCC photo ID cards, followed school sign-in 
procedures, and were accompanied by the test coordinator or TA to the testing room. The 
observers positioned themselves near the administration computer when possible and 
remained seated (as students arrived) throughout the test. They used applicable resources 
(excerpts from the administration manual or a job aid’) and entered observation ratings and 
notes in the observation checklist using either their laptops or paper copies of the observation 
checklist. 


Instruments. HumRRO developed two data collection instruments for use during 
administration observations. The first instrument contained a list of activities (“tasks”) 
performed by the TA that were based on the guidance for test administration in the PARCC Test 
Administrator Manual. Observers rated each task on a rating scale (described below). This 
instrument included space for notes to provide the rationale for the rating as well as other 
observations not captured by the rating scales. The second instrument, a semi-structured 
debriefing protocol, was developed for post-assessment interviews. All instruments were 
reviewed and approved by PARCC prior to their use. 


Test administration observation checklist. We designed the observation checklist as an internal 
data collection form, targeting specific test administration activities. The purpose of the form 
was to record whether specific administration tasks were conducted and to document 
exceptions to established procedures, if any were found. HumRRO used the PARCC Computer 
Based Testing (CBT) Test Coordinator, Administrator, Accessibility Features, and 
Accommodations manuals to identify the observable tasks included in the checklist. 


The observation checklist was an Excel® document that HumRRO observers used to enter data; 
data was uploaded to Statistical Analysis Software (SAS™) to facilitate aggregated reporting 
across observations. The full observation checklist can be found in Appendix A. The observers 
were trained to rate each checklist task on a five-point rating scale. The rating scale and a 
description of each scale anchor are presented below in Table 2. 


° This was a resource developed by HumRRO and PARCC staff containing the procedures used to administer 
accommodations. 
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Table 1. Checklist Observation Form Rating Scale 


Rating Description 


Met The task was completed 


Partially Met Some parts of the task were not completed 


1 

2 

3 Did Not Meet The observer knows that the task was not completed 

4 Not Observed The task was likely completed but was not directly seen by the observer 
5 


Not Applicable The task is not intended to be completed in certain situations 


In addition, to support the use of the observation checklist when observing accommodated 
sessions, HumRRO developed an accommodation job aid. Since the tasks the test 
administrators (TAs) follow vary for each testing accommodation that students require, as 
specified in their Individualized Education Program (IEP), listing all possible combinations in the 
checklist was not feasible. Observers used the job aid, based on PARCC accommodation 
documents (e.g., manual, training slides), to guide their ratings while observing an 
accommodation session. 


Test coordinator and administrator debriefing protocol. To obtain additional information 


regarding the overall test administration experience of the Test Administrator (TA), Test 
Coordinator (TC), and other school or district staff (e.g., other teachers helping the TA, 
technology personnel), we developed a debriefing form as a guide for conducting post- 
assessment interviews. If time permitted, HumRRO observers asked the TA or other staff about 
their experiences and feedback regarding the testing process, training materials, and support 
from PARCC. See Appendix B for the debriefing protocol. 


Participants. HumRRO sampled from the following states: Illinois, Maryland, New 
Mexico, Ohio, and Tennessee. PARCC provided HumRRO the contact information for a point of 
contact (POC) at each state department of education. Additional factors that applied when 
selecting and scheduling schools included: academic content area (ELA/Literacy and 
Mathematics), grade spans (elementary, middle, and high schools), and geographic locale. Our 
goal was to complete observations in 70 schools across the PBA and EOY computer/tablet- 
based administrations. 


School recruiting was a multistep process. Once states were identified, HumRRO received login 
permission to PearsonAccess™ which contained PARCC’s test registration information for each 
state. HumRRO only accessed the state (“organization”) extracts; this information included 
school names or codes, the testing planned at each school (PBA, EOY, or both), city names, and 
some district- or school-level contact information. HumRRO was not authorized to access any 
student-level information from PearsonAccess, nor did they have any interest in student-level 
information. We initially selected 25-30 schools in various cities, districts, or counties in each 
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state. We sent the list of schools we compiled to each state POC requesting permission to 
contact the schools. After obtaining state-level authorizations (typically a letter or reference), 
we contacted school districts requesting the testing schedule for each school we selected. After 
the school district verified our request, they generally referred us to specified school staff to 
obtain that information. After obtaining testing schedules we confirmed the visit with the 
school, provided schools a document introducing HumRRO, and reinforced our commitment to 
observe testing as unobtrusively as possible. Additionally, we assured schools that our findings 
would be reported in aggregate only and that we would focus on the administration of the 
PARCC field test for validity, quality assurance, and process improvement purposes. After this 
process, we obtained permission to observe 77 individual test sessions (e.g., grade 4 ELA 
session 3, grade 7 Mathematics session 2) in 61 schools. Table 1 provides descriptive 
information on the site observations. 


When discussing the notes made by the observers, throughout this report, the frequency of 
occurrence of those notes are reported by number of schools rather than by the number of 
sessions. This is because in every case, except one, the additional session observed at a school 
was the continuation of the test for the same students, in the same testing room, and 
administered by the same TA. 


Table 2. Summary of CBT Administration Observations 


Ke) e-) | Total oe) iat Ke) | 
Number of Sessions me) im Ko) e-) | Number of Schools Number of 
Sessions Observed Number of Schools Visited KYol eYe) (3 
(0) oY aN{-Xe| Sessions Visited Visited 
State 
Illinois 0 17 17 22.1 0 10 10 16.4 
Maryland 16 11 27 35.1 13 10 23 36.7 
New Mexico 6 8 14 18.2 6 6 12 19.7 
Ohio 7 3 10 13.0 7 2 9 14.8 
Tennessee 9 0 9 11.7 7 0 7 11.5 
Total 38 39 77 33 28 61 
Content area 
English/ 94 22 46 59.7 2 16 38 apne 
Language Arts 
Mathematics 14 17 31 40.3 11 12 23 37.7 
Total 38 39 77 33 28 61 
Grade level 
Elementary 17 20 37 48.1 16 14 30 49.2 
Middle 9 16 25 32.5 7 12 19 31.1 
High 12 3 15 19.5 10 2 12 19.7 
Total 38 39 77 33 28 61 


Note. Percentages may not add to 100 due to rounding. 
Total Number of sessions observed n = 77; total number of schools visited n = 61. 
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Study 2: Test Administrator Surveys 


Procedure. An online Test Administrator Survey was completed by TAs following 
administration of the PARCC PBA and/or EOY field test. The primary purpose of the survey was 
to collect information from the perspective of TAs on the effectiveness of the training they 
received and to identify potential problems that arose during test administration. The TAs were 
provided a web link to the online survey in their test administrator manual. They were 
instructed to log on and complete the survey after test administration was completed. If TAs 
administered both the PBA and the EOY, then they were instructed to complete the survey 
after the EOY administration. The TAs responded to a separate survey for computer/tablet- 
based test (CBT) administration and for paper-based test (PBT) administration. The online test 
administrator survey was available from March 24, 2014 to June 13, 2014, which corresponded 
to the first day of PARCC field test administration and to one week after the last day of PARCC 
field test administration. 


Instruments. HumRRO researchers developed a draft Test Administrator (TA) Survey 
based on an earlier version of the survey that HumRRO developed for the PARCC Item Try Out 
study in spring 2013 (Sinclair & Thacker, 2013). The draft survey was submitted by HumRRO to 
PARCC leadership for review and edits. The draft version covered the following topic areas: 


e potential issues and areas for improvement with regard to the training of TAs 

e potential issues and areas of improvement with regard to the administration protocol 

e potential sources of confusion for students 

e occurrence of technical difficulties 

e qualitative input on recommendations for improving administration policies and 
procedures 


The draft version was revised and edited based on PARCC leadership feedback to include survey 
items addressing these additional topic areas: 


e PARCC’s timing policy 

e use of accessibility features and accommodations 
e practice with sample items and tutorials 

e options for students after completion of testing 


Additional response scales were also added to the survey. A separate survey for TAs 
administering paper-based tests (PBTs) was also developed, as several survey items related to 
technology were not relevant to administrators administering PBTs. The items that appeared on 
the CBT version of the TA survey are presented in Appendix C and the items that appeared on 
the PBT version of the TA survey are presented in Appendix D. 
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The TA Surveys included a combination of close-ended survey items and open-ended survey 
items. Across the two TA Surveys there were 20 items that were open-ended or text box items. 
The purpose of the open-response data from these TA Surveys was to help identify concerns 
from field test that should be considered for operational testing. Given that there were several 
thousand TAs who responded to the surveys, every comment to every open-ended item was 
not individually coded. Rather, thousands of comments were read by analysts to identify 
recurrent themes for each open-ended item. Keywords were identified for each recurrent 
theme, and keyword searches were conducted on those keywords. This provided a good 
indication of the relative prevalence of the themes. However, because the intent of the field 
test surveys was to discover potential concerns, not to identify the frequency of each concern, 
coding of every comment on each open-ended item was not conducted. Consequently, 
throughout this report the common themes identified in the open-ended comments are 
reported, although the percentage of times that each theme was mentioned is not reported. 


The CBT Student Survey included an open-ended survey item asking students to describe 
anything that was confusing or unclear about the test. Across the four breakouts of the CBT 
Student Survey (i.e., PBA Math, PBA ELA, EOY Math, EOY ELA), approximately three-quarters of 
a million students responded to the CBT Student Survey. The purpose of the open-response 
data from this survey item was to help identify concerns from field test that should be 
considered for operational testing. Given that there were nearly three-quarters of a million 
students who responded to the CBT Student Surveys, every open-ended comment was not 
individually coded. Rather, thousands of comments were read by analysts to identify recurrent 
themes to the open-ended item. Keywords were identified for each recurrent theme, and 
keyword searches were conducted on those keywords. This provided a good indication of the 
relative prevalence of the themes. However, because the intent of the field test surveys was to 
discover potential concerns, not to identify the frequency of each concern, coding of every 
comment was not conducted. Consequently, throughout this report the common themes 
identified in the open-ended comments are reported, although the percentage of times that 
each theme was mentioned is not reported. 


Participants. There were 8,729 respondents who logged on to the TA Survey link. 
Respondents who only responded to the background survey items and/or who only responded 
to one or two other items on the survey were removed from the analysis. This resulted in 6,322 
respondents with 4,635 of those responding to the survey for CBT administration and 1,687 
responding to the survey for PBT administration. The response breakout by state and by mode 
of administration is presented in Table 3. 
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Table 3. TA Survey Response Breakout by State and by Mode of Assessment Administration 


Computer/tablet-based test Paper-based test 
Administration Administration 

State n % n % 

Arizona (N = 493) 295 59.8 198 40.2 
Arkansas (N = 471) 314 67.0 157 33.0 
Colorado (N = 223) 152 68.2 71 31.8 
District of Columbia (D.C.; N = 35) 31 89.0 4 11.0 
Florida (N = 8) 7 87.5 1 12.5 
Illinois (N = 1024) 737 72.0 287 28.0 
Louisiana (N = 311) 304 98.0 7 2.0 
Maryland (N = 1171) 710 60.6 461 39.4 
Massachusetts (N = 776) 601 774 175 22.6 
Mississippi (N = 148) 94 63.5 54 36.5 
New Jersey (N = 788) 785 99.6 3 0.4 
New Mexico (N = 48) 48 100.0 0 0.0 
New York (N = 57) 54 94.7 3 5.3 
Ohio (N = 393) 240 61.1 153 38.9 
Rhode Island (N = 183) 128 69.9 55 30.1 
Tennessee (N = 183) 128 69.9 55 30.1 
All States (N = 6322) 4635° 73.3 1687° 26.7 


Note. Percentages are based on the total number of valid responses for each state. Number of missing for CBT 
administration = 7. Number of missing for PBT administration = 3. 

* Includes the 7 cases that did not indicate their state. 

” Includes the 3 cases that did not indicate their state. 


Study 3: Student Surveys 


Procedure. Upon completion of the PBA and EOY field test, students completed a survey 
about their test-taking experience. Different surveys were administered following completion 
of Mathematics and ELA assessments. There was considerable overlap in the content of the 
items on the Mathematics and ELA surveys, although each contained a subset of items specific 
to the content area being assessed. There was no difference in the Student Survey for PBA and 
EOY, although the survey data was collected and reported separately for each. Similar to the TA 
Survey, a slightly edited version of the surveys were administered to students who were 
administered paper-based versions of the field test. 


Instruments. HumRRO researchers developed draft Student Surveys based on an earlier 
version of the Student Survey that HumRRO developed for the PARCC Item Try Out study in 
spring 2013 (Sinclair & Thacker, 2013).’° Separate draft surveys were developed for 
Mathematics and ELA, and for CBTs and PBTs. In addition, separate versions of the survey were 
developed for elementary students and for middle and high school students with the primary 
distinction being the response scale. The surveys for elementary grades contained a “Yes/No” 
response scale, and the surveys for middle and high school grades contained a four-point 


i During item try-outs in spring 2013, only students taking the ELA tests were surveyed. 
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agreement scale. The draft surveys were submitted by HumRRO to PARCC leadership for review 
and edits. The draft versions covered the following topic areas: 


e clarity of the directions read by the Test Administrator 

e clarity of the directions for questions on the test 

e helpfulness of the tutorial (practice questions) 

e whether the words on the test were easy to understand 

e whether the test questions covered topics already learned in school 

e ease of navigating through the test 

e usefulness of tools (CBT version only) 

e familiarity with using computers and keyboards 

e preference for computer-based or paper-based tests 

e occurrence of technical difficulties (CBT version only) 

e qualitative input on whether there was anything particularly confusing or unclear about 
the test (CBT version only) 


Based on feedback from PARCC leadership, minor revisions were made to the topics covered on 
the submitted draft Student Surveys, although the formats of the questions were edited 
considerably. Also, the separate surveys for elementary students and for middle and high 
school students were eliminated. Instead, the same surveys were administered to all grade 
levels, which included a variety of response scales (e.g., dichotomous yes/no response scales, 
multiple Likert-type rating scales, and select all that apply rating scales). In total, four versions 
of the Student Surveys were developed. The items that appeared on the Student Survey for the 
computer/tablet-based PARCC Mathematics field test (CBT math) are provided in Appendix E. 
The items that appeared on the Student Survey for the computer/tablet-based PARCC ELA field 
test (CBT ELA) are provided in Appendix F. The items that appeared on the Student Survey for 
the paper-based PARCC Mathematics field test (PBT math) are provided in Appendix G. Finally, 
the items that appeared on the Student Survey for the paper-based PARCC ELA field test (PBT 
ELA) are provided in Appendix H. 


Participants. Participants were the students completing the PARCC field test. Table 4 
below identifies the number of cases (i.e., students) contained in the initial raw data files’ 
provided by Pearson as compared to the number of cases for which there was valid survey 
data. Valid survey data was defined as non-duplicate cases for which there was a response to at 
least one item on the survey. In other words, duplicate cases’’—i.e., those that had the same 


** Raw data files for Student Surveys included individual student responses to each question on the student survey. 
Student IDs and test IDs were included in the raw data file, but HumRRO had no way of matching student names to 
Student IDs. Student ID and test codes were stripped from the data file prior to running analyses. 

- Duplicate cases for the same students were possible on the CBTs if a test was submitted, un-submitted, 
restarted, and then resubmitted again. Personal communication with Pearson analyst on August 6, 2014. 
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student ID, the same test ID, and the same survey responses—and cases with all missing data 
were removed from the analysis file. The information presented in Table 4 indicates that, of 
those who took the PARCC field test, approximately 85% - 90% responded to one or more items 
on their Student Survey. 


Table 4. Student Survey Response Rates 


Number of Cases in Number of Valid Cases Percentage of Valid 
Initial Raw Data File in Analysis File Cases in Raw Data File 
Test Type n n % 
CBT 
PBA Math 207,966 183,099 88.04 
PBA ELA 252,602 221,158 87.55 
EOY Math 180,731 157,872 87.35 
EOY ELA 180,606 160,820 89.04 
PBT 
PBA Math 81,644 71,452 87.52 
PBA ELA 77,322 65,573 84.81 
EOY Math 70,573 60,073 85.12 
EOY ELA 60,477 53,279 88.10 


Note. CBT = Computer/tablet-based Test administration; PBT = Paper-based Test administration; 
PBA = Performance-based Assessment; EOY = End-of-Year; ELA = English Language Arts. 


Results 


The findings from the three studies are presented in this section according to the research 
question they address. It is should be noted that there are several instances of overlap among 
the research questions. For example, Research Question 7 (Are there any interruptions during 
the session?) overlaps with Research Question 4 (To what degree are students engaged in 
taking the test? (i.e., interruptions often negatively impact engagement)). Consequently, one 
set of findings may be relevant to multiple research questions. Thus, there is some redundancy 
in the reporting of results across research questions; however, in order to reduce the length of 
the report some results are presented for one research question, but are identified as being 
relevant to other research questions. Where applicable, references are provided to the actions 
that PARCC is taking to address these research findings; the PARCC actions come from the 
PARCC Field Test: Lessons Learned report (PARCC, 2014).*° 


The full set of item-level results for the TA Surveys (with breakouts by state) can be found in 
Appendix | (CBT version) and Appendix J (PBT version). The full set of item-level results for the 
Student Surveys (with breakouts by state, and by PBA and EOY) can be found in: Appendix K 
(CBT Math version), Appendix L (PBT Math version), Appendix M (CBT ELA version), and 
Appendix N (PBT ELA version). 


= http://www.parcconline.org/parcc-states-release-lessons-learned-report 
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Findings for Research Question 1: To what degree do the TAs find the instructions clear, 
sufficiently detailed, and easy to follow? 


The claim in the theory of action most directly related to this research question is, “Teachers 
have the resources and supports to administer the assessments.” The instructions in the Test 
Administrator Manual serve as the TAs’ primary resource for administering the assessments. An 
important assumption underlying this claim is that this resource (i.e., the TA Manual) is 
sufficiently clear and detailed, and easy to implement. To verify the veracity of this assumption 
the TA Surveys asked TAs to rate their agreement with the statements, “The policies and 
procedures within the Test Administrator Manual were easy for me to understand,” and “The 
instructions (including the scripts) within the Test Administrator Manual were easy for me to 
implement.” The findings presented in Figure 3 indicate that more than two-thirds of the 
administrators agreed that the policies and procedures in the TA Manual were easy for them to 
understand. Slightly fewer (55% - 63% for PBT and CBT, respectively) agreed that the 
instructions (including the scripts) in the TA Manual were easy for them to implement (see 


Figure 4). 
The policies and procedures within the Test Administrator 
Manual were easy for me to understand. 

100% 
90% 
0, 

as 67.20% 
‘0 
% 60% 

_ 
& 50% 
x 40% 
30% 
20% 
10% 
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Computer-based Administration Paper-based Administration 
n= 4,635 n= 1,687 


Mode of Administration 


Figure 3. Percentage of TAs agreeing that TA Manual is easy to understand. 

Percentages are based on the number of valid responses. Missing = 223 for CBT administration and missing = 16 
for PBT administration. Ratings made on a 4-point agreement scale where 1 = Strongly Disagree, 2 = Disagree, 

3 = Agree, and 4 = Strongly Agree. Results reported here reflect the combined percentage of “Agree” and “Strongly 
Agree.” 


Updated February 17, 2015 Page 15 


PARCC Quality of Test Administration 


The instructions (including scripts for administering the 
assessment) within the Test Administrator Manual were easy 
for me to implement. 
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Figure 4. Percentage of TAs agreeing that instructions in the TA Manual were easy to 


implement. 

Percentages are based on the number of valid responses. Missing = 225 for CBT administration and missing = 19 
for PBT administration. Ratings made on a 4-point agreement scale where 1 = Strongly Disagree, 2 = Disagree, 

3 = Agree, and 4 = Strongly Agree. Results reported here reflect the combined percentage of “Agree” and “Strongly 
Agree.” 


Although the instructions were largely rated as easy to implement, open-ended comments from 
the TA Surveys provide some insights into sources of confusion with instructions in the TA 


14 
F 


Manual.” Several themes emerged in the TAs’ comments on the CBT version of the TA Survey. 


Those themes and example comments’ illustrating the themes are: 


Confusion caused by skipping parts of instructions that were not relevant: 


e The direction manual was very difficult to follow along with. Directions should be 
separated by grade level for easier teacher reading and less confusion or mistakes. 

e It was confusing to have to skip large sections at a time because they did not apply to my 
grade level. It made it very easy to get lost in the script. 


“These open-ended comments are also relevant to Research Question 3, “Do students appear to understand the 
instructions provided to them.” 
» All survey comments in this report are provided unedited, exactly as they were entered by TAs and students. 
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The instructions should be separated by grade level. It was confusing to have to jump 
from box to box, skipping over 3-5 to get to 6-8, etc. 

Manual was confusing at first because directions were not specific to the actual grade 
level; they were very generic. | had to search the manual for the times of each section. 
There should be "tabs" or separate sections for the different grade levels that are more 
specific so the company can ensure that the directions are truely being given the same 
way no matter where the test is given -- take out part of the human error from the 
directions. Very specific directions would be better! 


Confusion over number of test items and time to complete items: 


The timing of the test was confusing. | tried to make it as simple and clear as possible for 
students but it was difficult since | was having difficulty understanding it. Example: The 
time chart on pg. 27 of manual couldn't be completed because the test never indicated 
how many questions were in each section or how much time a section could take. 

The script for additional time vs. regular time was a bit confusing. Also the script for the 
save and return option at the end of the session was confusing. 

it was really confusing that the exact testing time, number per session, etc., was not 
right along with the directions for me to see. | had to go back and skim through the book 
as | administered which was a time waster. The manual was not user-friendly at all. 

The one thing that | thought was confusing is that | didn't think the directions were clear 
in how many questions the students would answer. 

It was unclear how many sessions we would do in a day. The students finished session 
one in about a third of the allotted time. Because there was so much time left, it was 
assumed that there must be a few sessions to be given on that day. So my students 
completed the three days of testing in one morning. Ending a section with a green 
button that said "Continue" was very confusing. We were also given 9 different seal 
codes so we thought that there would be three sessions a day. 

When the children logged in, it said there would be 3 parts with 41 questions, but it was 
unclear if that was all for the first day or for the entire test over the two days. Students 
(and teachers) wanted to know how it would be broken down. In the testing we are used 
to administering the script will say, "Today you will take Part 1. There are 13 questions in 
this part," or "Today you will take 3 parts. The first part has 8 questions." 

The Seal Code instructions were not clear. When administering the test, because there 
were 9 Seal codes offered, it was unclear (and | did not find any instructions within the 
manual) whether Seal Codes were used WITHIN Section 1 so all 9 Seal codes could be 
used or to move from one full section to the other, which is the correct way. My Students 
moved forward to Session 2 before everyone was finished with Session 1. Therefore 
Session 2 has random start times. | also would have been helpful to know the number of 
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questions in each section. It did not seem right that there were only 11 question ina 
section and they needed 80 + minutes... Another reason | was confused about the use of 
Seal Codes. | felt it should have taken longer to complete one section given the time 
therefore we used the Seal Codes too early. | get it now! but would have been more 
helpful to practice this during the infrastructure trial. 

Providing the additional time beyond the regular session time was confusing because 
students weren't sure whether they were supposed to try to answer all questions by the 
end of the original given time or not. 


Confusion over “Section” vs. “Session”: 


In the testing manual, the words are intertwined; sessions, sections, tests, etc. It is 
confusing. Too many different places for references in the various manuals and they 
aren't always the same directions. 

There was some confusion because the field test said there were four sessions on the 
students computer direction page, they were including the survey and that was 
confusing for students. 

The use of 'section' and 'session' was confusing. 

The tests are 'sessions', but the directions say 'sections' 


Confusion caused by inconsistencies between what is in the script and what is on the screen: 


The script has the proctor tell the students to click on the start button and then continue 
reading directions. This means that the first test question was visible on the screen 
BEFORE the students were allowed to begin testing. This confused some students. The 
script does not accurately describe the buttons students should click. For example, the 
directions instructed students to click on the blue button in the middle of the screen, 
when in fact the button was in the upper right-hand quadrant. 

The instructions did not match the screen the students were looking at while | read them. 
The instructions for ending a session and finish later were confusing for students and 
some clicked the wrong one and it completed their test and they still had 3 sessions to 
take. 

Too many buttons to press or not press. Confusion about buttons that say ‘continue’, but 
students are not supposed to press them, until further directions are given. Buttons 
about submitting final answers: If students are not supposed to press them, then why 
have it as an option??? It should only be an option for the last test students take. During 
test 2 and 3 reference to blue buttons is sometimes omitted in manual...| used the 
directions from day 1. 

When the students first log in, they are sent to a welcome screen. In the script, it never 
tells the students to click on the option to begin the test. Therefore, when you tell the 
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students to enter the seal codes, they are not on the right screen. This was a little 
confusing. | also found the instructions confusing for the students who are taking 
extended time, or not requiring any at all. | had to backtrack a little to figure out what 
part | was supposed to be reading to them. 

In the directions, it stated that students should NOT press the green continue button 
after each section. However, when students reached this page after testing, it told them 
to do one of two things, one of which was to hit the green continue button to go to the 
next session. This confused students because | told them in the directions not to touch it 
but then they were given the option to click on it. 


Confusion about navigating through the test: 


The directions were somewhat unclear about when to click continue and move on to 
future screens. There were many questions throughout the test if they were to go on or 
not. It would be nice if there were directions on the screen for students when they finish 
a portion that says, please continue or stop. 

At the end of a session, there should be a STOP button, not a CONTINUE button as this 
caused great confusion with the students and test administrators. 

The instructions to stop on the green "continue" button were confusing because during 
Algebra 2 session 1, students were actually supposed to continue from the non- 
calculator to the calculator part of the test. 

There was confusion on day one regarding submitting answers, completing tests, and 
how many seal codes were necessary. 


Open-ended comments from the PBT version of the TA Survey indicate that, as with the CBT 
administration, some TAs administering paper-based tests reported that the instructions were 


confusing with regard to: additional time, skipping over instructions that were irrelevant to the 


grade being tested, and disconnects between what appeared in the script and what appeared in 
students’ test booklets. There were some additional themes that emerged that were unique to 
PBT administration. Those themes and example comments illustrating those themes are 


presented below. 


Confusion caused by no page numbers: 


Students were confused without page numbers in the instructions. Students found 
navigating between two books confusing. 

Some students wondered why | didn't tell them the exact page and question number to 
begin on. Some wanted to help me out by telling the class what page we were on, but 
not all students were on the same page! | know they are used to different Test Forms 
from previous standardized tests, but the different page numbers confused some of 
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them. | would suggest that blank pages be inserted as necessary so all students are 
basically on the same page at the same time. 

e When starting a new session it was hard to find the page to start since there were no 
page numbers in the directions to direct students too. While monitoring the test it was 
hard to determine if the students were on the right section because it did not indicate on 
the top of the page which section students were working on. 

e Page numbers were not provided for students to find starting page for each session. 
Instructions did not explain how to find proper spot in answer booklet, it seemed to be 
implied from instructions for Test booklet. Students were confused and were not ready to 
start the test after instructions were finished. 

e Students were confused that | was unable to give them a page number to turn to in the 
answer document and test booklet. 


Confusion about calculator use: 


e Additionally, there was confusion, both while reading instructions and during the test, 
about the directions for the distribution of calculators during session 

e [twas very confusing to the students as to when they were able to use their calculators 
on session 1. In the directions | read to them, the script, it stated they couldn't use their 
calculators at all and then later on stated their were some problems they were able to 
use their calculators. 

e Allowing calculators for part of one test is impossible to monitor. 

e Unsure when they could or could not use calculators. 

e During part 1 of session 1, it was confusing to them and to me with the use of 
calculators. They were allowed during part of part 1 but not during the entire time. The 
second session went more smoothly since they had calculators during the entire session. 
Perhaps the test could be set up so that there is a cutoff to give calculators all at once to 
students. 


Next, findings from the observation study also provide valuable insight about sources of 
confusion with instructions in the TA Manual. Concerns about the TA script surfaced through 
the researchers’ observation notes and from the feedback provided during the debriefing 
interviews. Issues noted by the researchers conducting the observations included the script’s 
lengthiness and ambiguous terminology. First, the observers noted that some TAs had difficulty 
following the script because of its lengthiness. Consistent with feedback on lengthiness, 
observer notes from 11 schools indicated that modifications were made while reading the 
script to adapt to specific testing conditions, such as back-to-back sessions or to address 
common issues that arose during previous testing sessions. The most common modification 
made by TAs (noted in five school observations) was skipping repetitive parts of the script when 
starting subsequent sessions. Another modification noted in three schools was to provide 
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additional guidance on logging into TestNav or on use of headphones. Typical observer notes 
were, “Modified script to direct students through Java issues,” “... (need to state) go to the URL 
on the ticket,” and “TA added note about headphones before beginning the script.” 


Observer notes from six schools indicate that there was confusion caused by ambiguous 
terminology in the TA scripts. For example, observers noted the following: “...directions tell 
students to click the ‘start’ button, but the screen has a ‘resume’ button,” “TA had to clarify 


nt 


that the ‘testing device’ is the computer,” “change the session word to mean part at times. You 
can continue to the second part of the session (page 113) but what does session mean this 
time?” Also, observers in six schools indicated that there was ambiguity in the instructions for 
ending sessions and for managing the session time during the mathematics fluency portion of 
the assessment. Some of the comments from observers were, “The TC and TA had difficulty 


nt 


interpreting the script’s instructions for ending the session,” “...students needing more time 
had to be told over and over not to do anything and not to follow the instructions (the TA was 
reading for the students who were finished),” “ The TA pointed out in one place the survey was 
referred to as the ‘survey’...and in another place referred to it as a third ‘session,’” and “...the 
fluency questions messed up the timing because students were told to stop and the script was 


not clear enough (i.e., that there were more items).” 


In addition to observer notes, there were comments and suggestions from 25 schools obtained 
during the debriefing interviews in reference to the organization of the script. Findings from the 
debriefing interview, revealed that some TAs did not like using one script for all grades and 
testing situations (i.e., one session per day vs. consecutive test sessions within the same day), 
and in some cases this was a source of confusion. Skipping the sections TAs did not need and 
finding the sections they did need was sometimes difficult as they administered the 
assessment; therefore, the TAs indicated that they would like a streamlined script that they 
could read from start to finish. Typical comments were, “...confusing to skip across sections that 
were not relevant to her grade level,” “...need to streamline (scripts) to only what has to be said 


yD lt 


and is said only once and targets the grade...,” “...a lot of directions for other grades that had to 
be skipped, which required a lot of page flipping,” and “Students need more visual instructions 
instead of the wordy script.” Moreover, eleven schools described confusion with the script 
when they started or stopped a session. A suggestion repeated by several TAs during the 
debriefing interviews was that an overview of the procedures in a bulleted list with the time 


chart would be helpful. 


Furthermore, the comments received during the debriefing interviews at 18 schools provided 
additional support to the observer notes above regarding confusion with navigating through 
the test. TAs described initiating and closing sessions as confusing. For example, a suggestion 
received from two different schools was to mirror the icons students are accustomed to seeing 
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on paper and pencil tests such as a stop sign, rather than displaying a green button at the end 
of a session, which students misinterpreted as a signal to continue. Other TAs suggested 
clarifying and minimizing the steps necessary to close sessions. 


Although most comments described concerns with the scripts, comments received from 19 
schools indicated that the administration manuals were cumbersome to use and contained 
information that was not relevant for their role in testing. Typical comments were, “too many 


YD ll, 


manuals, too long,” “would like a summary page or checklist of major steps | have to do,” 


DMM 


“..would also like computer screen captures since | am a visual person,” “couldn’t easily find 
the information regarding of the number of questions on each test session...and timing,” and 
“better to have chapter summaries.” On the other hand, comments received from eight other 
schools indicated the administration manuals and materials were appropriate and helpful. One 
TA commented that the best part (of the materials) was the computer set-up information 


(Infrastructure Readiness Guide), which was “well written, organized, and step-by-step.” 


In summary, overall, more than two-thirds of TAs agreed that the policies and procedures in the 
TA Manual were easy for them to understand. Slightly fewer agreed that the instructions in the 
TA Manual were easy for them to implement. Open-ended feedback from the TA Surveys and 
from the debriefing interviews indicates that some TAs encountered confusion with the TA 
Manual. The TAs commonly indicated that the TA Manual was poorly organized (i.e., inclusion 
of material that was not relevant to the grade being tested made it overly cumbersome and 
difficult to find relevant sections), contained ambiguous terminology (e.g., “sections” and 
“sessions”), and had disconnects between what appeared in the script and what appeared on 
students’ tests. These findings suggest that some improvements are needed to enhance the 
clarity of the TA Manual. Improvements to the clarity of the instructions in the TA Manual 
should help to ensure that the test is administered in a standardized manner by all TAs, which 
will thereby tighten adherence to Standard 4.15. Actions that PARCC is taking to address these 
concerns are provided in the Executive Summary and in the Summary and Recommendations 
sections under Claim 2. 


Findings for Research Question 2: To what degree do the TAs follow the protocols and 
instructions? 


The claim in the theory of action most directly related to this research question is, 
“Administrators are prepared to administer the assessments as intended.” For this claim to be 
true at least two assumptions must be met: (a) administrators must be trained on how to 
administer the assessments, and (b) that training must be effective. To investigate the veracity 
of these assumptions, the TA Survey (CBT version) asked the TAs a series of questions about the 
training they received prior to administering the assessments. First, TAs were asked to indicate 
if they completed any of the PARCC online training modules. Nearly, three-fourths (74%) of the 
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TAs indicated that they had completed one or more of the online PARCC training modules. 
Next, those indicating they had completed online training module(s) were asked to identify 
which modules they had completed. The findings displayed in Figure 5 indicate that more than 
three-fourths completed the Computer-Based Test Administration module. In other words, just 
over half of all the TAs responding to the TA Survey (for CBT) completed the Computer-Based 
Test Administration module. Furthermore, of those who indicated they completed online 
training only a small percentage of TAs (16% or less) reported that they completed any of the 
other training modules.*° 


*® While not included in the results presented in Figure 5, the TAs were also asked to indicate if they completed the 
online training module for “Accessibility Features and Accommodations with Computer-based Testing.” Those 
results are reported in a separate, subsequent section titled, “Findings for Research Question 2 Specific to 
Accessibility Features and Accommodations.” 
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% Selected a Module 


Of those who completed online training modules, which modules did they complete? 
(select all that apply): 


100% 
90% 
80% 
70% 
60% 
50% 


40% m= Computer-based 
Administration 
0, 
ane n= 3,422 


16.10% 


20% 13% 12.10% 


10% 
0% 


Computer-based Testing Setting Up an Technology Setup Student Data Upload Emerging Technologies 
Test Administration Infrastructure Overview and Security with 
Trial/Dress Rehearsal Computer-based Testing 


Figure 5. Of the TAs indicating completion of online training, percentage who completed each module. 

TAs who did not select any of the online modules were excluded from the results presented here. 

The TAs were also asked about completion of online training on Accessibility Features and Accommodations with Computer-based Testing; those results are 
reported in a separate, subsequent section on accessibility features and accommodations. 
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The TAs were also asked to indicate whether the online trainings were effective at informing 
them of the protocol to follow when administering the assessments, and whether the training 
had effectively prepared them to resolve basic problems related to technology (e.g., logging 
students in, exiting the test, etc.). Those results are presented in Figure 6, and indicate that 
three quarters of the TAs (75%) agreed that the online training effectively informed them of the 
protocol to follow when administering assessments, and slightly more than half (53%) agreed 
that the online training prepared them to resolve basic problems related to technology. 


Of those who completed online training, did they find it 


effective? 
100% 


90% 


75.20% 


80% 
70% 


60% 53.30% 


50% 


% Agree 


40% m Computer-based 


Administration 


0, 
au% n= 3,422 


20% 
10% 
0% 


PARCC online training modules PARCC online trainings prepared 
effectively informed me of me to resolve basic problems 
protocol to follow when related to technology 
administering assessments 


Figure 6. Percentage of TAs agreeing that online training was effective. 

Percentages are based on the number of valid responses. Ratings made on a 5-point scale where 1 = Not 
Applicable, 2 = Strongly Disagree, 3 = Disagree, 4 = Agree, and 5 = Strongly Agree. For the percentages reported in 
this figure, “Not Applicable” responses were recoded as Missing. Results reported here reflect the combined 
percentage of “Agree” and “Strongly Agree.” 


In addition to inquiring about the online training modules, the TAs (both those administering 
the CBT and the PBT) were asked to indicate if they had received any training from their state, 
district, or school. The results in Figure 7 indicate that most had received training from their 
school, particularly those administering the PBT. Less than half of those administering the CBT 
had received training from their district and less than a third of those administering the PBT had 
received training from their district. Very few TAs indicated they had received training from 
their state, regardless of whether they were administering the CBT or the PBT. 
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Did you receive training from your state, district, or school? 
(select all that apply): 


100% 
90% 
80% 
70% 
60% 54.80% 


= Computer-based Administration 
n = 3,736 


32.80% m Paper-based Administration 
n= 1,472 


50% 
40% 
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20% 
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0% 
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Figure 7. Percentage of TAs receiving training from state, district, or school. 
Those who selected, “No, | did not receive any training from my state, district, or school” were removed from the 
analyses. 


The survey for TAs administering the CBT included an open-ended item asking TAs for 
recommendations on how PARCC can improve its online training. Several themes were 
identified from the TAs’ comments. The TAs recommended that training could be improved 
by:’” (a) clarifying procedures for logging on and off, (b) clarifying instructions for additional 
time, (c) improving navigability of the TA Manual by reducing the amount of ‘if this, then go to 
page X’, (d) including more training on how to use the tools, (e) condensing the amount of 
training materials by reducing redundant information and highlighting major responsibilities of 
TAs, and (f) making the TestNav system more user-friendly (e.g., by making it easier to 
troubleshoot technical problems). Another commonly mentioned recommendation was to 
provide in-person training. 


The findings from the observation study provide evidence that both supports and expands upon 
the findings from the TA Surveys. The observation study directly addresses the degree to which 
the TAs followed the administration protocols and instructions. First, the observation checklist 


” Note that some of the TAs’ suggestions for improving training dealt more explicitly with recommendations for 
changing the materials used by TAs (e.g., the TA Manual) and making improvements to the technology (e.g., 
TestNav). 
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includes a set of “pre-assessment activities” that, per the Test Administration Manual, TAs were 
to engage in prior to students beginning testing. The findings reported in Table 5 (the summary 
of pre-assessment tasks), indicates that most pre-assessment tasks (activities) were completed 
as intended with three exceptions.*® The first exception dealt with spacing between students 
(which has implications for cheating). Observer notes from 20 schools (26 observations) 
indicate that the primary issue with room preparation was the lack of use of partitions or the 
lack of sufficient space between computers. Typical observer comments were, “not enough 
seats (space) to put extra space,” and “students were close together but accustomed to it.” The 
second exception was posting of the “Testing in Progress” sign. In nearly a fourth of the 
observed sessions, the TAs did not post the sign. There were no observer notes of significance 
to indicate anything other than TAs neglecting to post the sign (such as significant interruptions 
caused by neglecting to post the sign) so this should be easily remedied in the future. The third 
exception, “Drawing a timing box on the board,” was either partially met or not met in about 
half of the observations. Observer notes reveal that: (a) a chalk or white board was not 
available at four schools (6 observations), (b) TAs used verbal updates only at six schools (7 
observations), and (c) TAs quit using the timing box at three schools (3 observations) because 
students finished early or connectivity issues made it challenging to keep track. Another eight 
schools (11 observations) did not use the timing box exactly as described in the manual and 
received “Partially Met” ratings. The findings reported above under Research Question 1 might 
help explain why many of the TAs did not follow the protocol of drawing a timing box on the 
board. Open-ended comments from the TA Surveys revealed that some TAs struggled to 
understand the instructions in the TA Manual for determining testing time. Consequently, if the 
TAs could not determine the number of items and the time to complete those items, this might 
explain why they did not use a timing box. 


Table 5. Summary of Pre-Assessment Observation Checklist Tasks* 


Partially Did Not Not 
Met Met Meet Observed NA 

Test Administrator Tasks % % % % % 
Prepare student computers for testing 
(close browsers and programs, launch 63.6 14.3 5.2 14.3 2.6 
TestNav). 
Prepare Test Administrator computer. 59.7 1.3 2.6 33.8 2.6 
Prepare testing room (every other seat 
or partitioned, room configured for 63.6 15.6 18.2 1.3 1.3 
security/privacy). 
Post "Testing - Do Not Disturb" sign on 51.9 39 32.1 30.8 1.3 


outside door(s). 
(continued) 


*8 It is important to note that a “Not Observed” rating only indicates that the observer did not see the task 
completed; therefore, in many cases the task was completed. 
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Table 5. Summary of Pre-Assessment Observation Checklist Tasks* (continued) 


Partially Did Not Not 
Met Met Meet Observed NA 
Test Administrator Tasks % % % % % 

Ensure prohibited materials are not 81.8 0.0 16 0.0 15.6 
visible. 
Confirm correct Student Authorization 80.5 0.0 0.0 18.2 13 
Tickets and seal codes for session. 
Draw a timing box on the board as 46.8 15.6 37.7 0.0 0.0 
described in script. 
Allow authorized visitors only. 83.1 0.0 1.3 0.0 15.6 


Note. NA= Not applicable. Number of observed testing sessions is n = 77. 
*Only CBT administrations were observed. 


Next, the observer findings from tasks completed during the actual test administration are 
summarized in Table 6. In addition to the timing box issues discussed above, another notable 
exception to following the administration protocol and instructions was reading the script 
verbatim. The task for reading the script verbatim was only met for approximately half of the 
sessions. This is an important task that helps to ensure consistency throughout PARCC test 
administrations. Observer notes indicate that the most common modifications were to skip 
repetitive parts of the script, change words (e.g., change “session” to “part”), and provide 
additional guidance (e.g., for logging into TestNav or on use of headphones). As discussed 
above under the findings for Research Question 1, several concerns about the script surfaced 
during the observations and during the debriefing interviews, including reports that the scripts 
were too lengthy and included parts that did not apply to the test takers, that some terms in 
the script were confusing (e.g., “session” vs. “section”), that the instructions on how to end 
sessions, how to manage additional time, and how to instruct students on the mathematics 
fluency assessment were difficult to understand, and that some of the information in the 
manuals was difficult to find when needed, such as the number of items per test and time limits 
for each test. These findings help explain why observers noted that scripts were followed 
verbatim in only about half of the observed sessions. 


Another deviation from protocol that was noted in approximately 20% of the observed sessions 
was that TAs did not meet or only partially met the requirement for monitoring student 
performance using the administration computer. The observer notes reveal that in six schools 
the administration computer was primarily used to reset students for connectivity problems 
and rarely used to monitor student progress. And, in at least three schools, the administration 
computer was not located in the room where students were testing. 
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Table 6. Summary of Assessment Observation Checklist Items 


Ratings 
Partially Did Not Not 
Met Met Meet Observed NA 
(%) (%) (%) (%) (%) 
Seat students at a computer with non- 96.1 26 0.0 13 0.0 
permitted materials removed. 
Fill in timing box when script advises. 57.1 18.2 24.7 0.0 0.0 
Provide students with the following items, as appropriate: 
a. Student Authorization Ticket. 97.4 0.0 0.0 1.3 1.3 
b. Scratch paper and pencil. Print name 675 143 143 13 16 


in upper right-hand corner 
c. Hand-held calculators. 23.4 1.3 3.9 1.3 70.1 
d. Tracing paper, reflection tools, 


; 5.2 0.0 3.9 1.3 89.6 
straight edge, compass. 

e. Headphones. 33.8 10.4 2.6 2.6 50.6 
Read script verbatim. 50.6 29.9 14.3 5.2 0.0 
Keep track of Hine during the test as 51.9 18.6 18.2 0.0 13 
noted in the script. 

Ensure students log in and have access to 98.7 0.0 0.0 0.0 13 
the test. 
Monitor student progress using 74.0 6.5 13.0 5.2 13 
administration computer. 
Monitor students by circulating around 97.4 26 H0 0.0 0.0 
the room. 
Answer student questions following 62.3 6.5 39 5.2 321 
script guidance. 

F é Ae ts 
Provide up to 50% additional time if 48.1 13 6.5 13 42.9 
needed. 
Provide a group 3 minute break, if 16.9 0.0 13 6.0 81.8 
necessary. 
eee test security if test is 6.5 0.0 13 0.0 92.2 
interrupted. 
Handle testing irregularities.* 23.4 1.3 0.0 0.0 75.3 
Handle asueenrs finishing early 87.0 6.5 16 6.0 36 
appropriately. 
Ensure students receiving the same 16 0.0 6 0.0 94.8 


accommodations are grouped together. 
Note. NA= Not applicable. Number of observed testing sessions is n = 77. 
* Respond appropriately in cases of loss of internet connectivity, safety threat, fire drill, or student cheating. 
° Follows state level guidance as documented in the PARCC Test Administrator Manual. 


Table 7 provides the observer rating summaries for the post-assessment tasks. Findings indicate 
that most post-assessment tasks were completed as intended, with two exceptions. First, 
observers did not often observe test materials being returned to the test coordinator because 
in 14 schools (17 observations) the same person filled both roles, and in four other schools the 
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testing went directly into another test session. Second, observers typically did not observe TAs 
completing the TA Survey. Although it is not clear why observers rarely saw the TA complete 
the survey, it is likely because teachers were returning to their classrooms and completing it 
later. In three schools, observers asked if the TA was planning to complete the survey. One TA 
indicated he or she would complete it later and two others indicated that they were not aware 
of the survey. 


Table 7. Summary of Post-Assessment Observation Checklist Items 


Ratings 
Partially Did Not Not 
Met Met Meet Observed NA 
% % % % % 

Students log off of the session. 88.3 3.9 2.6 1.3 3.9 
Collect Student Authorization Ticket and 38.3 39 13 16 3.9 
any test materials. 
Dismiss students in accordance with 38.3 16 13 26 5.2 
school procedures. 
Return testing materials to the Test 48.1 0.0 13 53.4 573 


Coordinator. 
Completes the test administrator survey. 1.3 0.0 1.3 72.7 24.7 


Note. NA= Not applicable. 
Number of observed testing sessions is n = 77. 


In addition to the findings from the observation checklist presented in Tables 5 -7, findings from 
the debriefing interviews provide additional contextual information to help explain the degree 
to which TAs followed protocols and instructions. Findings from the debriefing interviews reveal 
that TAs at eight schools indicated they were unaware of the training modules, videos, and 
practices tests; however, comments from 32 other schools referenced using the training 
resources. Comments received from 14 of those 32 schools described the training materials as 
helpful, while 7 schools indicated the training for TAs and students was not particularly helpful. 
Eight schools commented that the district-level training they received was the most beneficial. 
Finally, 13 schools suggested providing students and TAs with severa 


|" 


mock” testing programs 
so both students and TAs can practice accessing and navigating TestNav; they also expressed 
that gaining experience with all item types prior to test administration would be highly 
beneficial. 


In summary, training is an important component for ensuring that TAs carefully follow 
standardized procedures for test administration (AERA, APA, NCME, 2014, p. 114). Findings 
from the research studies indicate that only about half of the TAs who responded to the survey 
for CBT (which was 78% of those who indicated that they completed one or more of the on-line 
training modules) completed the training module on Computer-based Test Administration. Of 
those who completed one or more training modules, three quarters agreed that the training 
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was effective for informing them of the protocol to follow when administering assessments and 
slightly more than half agreed that the training prepared them to resolve basic problems with 
technology. For paper-based test (PBT) administration, TAs most commonly indicated that they 
received training from their school, but even then, more than a fourth indicated that they did 
not participate in any school-provided training on PBT administration. These findings suggest 
that a substantive number of TAs might not have received training on administering the PARCC 
field test, and for those who did, there may have been deficiencies in the effectiveness of the 
training for some. Findings from the observation study indicate that, for the most part, TAs did 
follow standardized procedures during test administration; however, there were a few 
exceptions. The most notable exception was that only about half of the TAs in the observed 
sessions read the administration script verbatim. The lack of adherence to the script might 
partly be due to training deficiencies. Deviations from the script might also be partly due to a 
lack of clarity in the TA Manual, as discussed under the results for Research Question 1. Also, 
TAs were allowed to clarify directions after the script was read verbatim. Greater adherence to 
the standardized test administration procedures, through such means as greater participation 
in training, improved training, and improved clarity of the TA manual, should help strengthen 
adherence to Standard 6.1. Actions that PARCC is taking to address these concerns are provided 
in the Executive Summary and in the Summary and Recommendations sections under Claim 1. 


Findings for Research Question 2 Specific to Accessibility Features and Accommodations 


Both the TA Surveys and the Observation Checklist included a section specific to the PARCC 
guidance on accessibility features and accommodations. The TA Survey (CBT version) asked TAs 
if they completed the PARCC online training module on, “Accessibility Features and 
Accommodations with Computer-based Testing.” Less than a third (30%) indicated that they 
had completed this training. The TA Surveys (both CBT and PBT versions) also asked if TAs had 
received training from their school or district on administering accessibility features or 
accommodations. Less than half of the TAs indicated that they had received this kind of 
training, regardless of CBT or PBT administration (see Figure 8). 
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Which types of training on ‘Administering Accessibility 
Features and Accommodations’ did you complete? 
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50% 
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Percent Selected 'Yes' 


29.90% 
30% 


20% 


10% 
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Online PARCC training module School or district training, CBT School or district training, PBT 
on Accessibility Features and Administration n = 4,635 Administration n = 1,687 
Accommodations with 
Computer-Based Testing, CBT 
Administration 
n=4,635 


Figure 8. Percentage of TAs indicating they received various types of training on Accessibility 
Features and Accommodations. 
CBT = Computer/tablet-based test; PBT = Paper-based test. 


The TA Surveys also asked TAs to indicate if they were aware that there is an approved PARCC 
Accessibility Features and Accommodations Manual and if they had read it. Those results 
indicate that only about two-thirds of TAs were aware of this manual, and, of those, only about 
two-thirds actually read the manual (see Figure 9). Relatedly, those administering the CBT were 
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asked if they were aware that there is approved PARCC guidance on “Computer-based 
Accessibility Features and Embedded Accommodations” and if they had read that guidance. 
Only slightly more than half of the TAs administering CBTs indicated that they were aware of 
this guidance, and, of those, nearly three-fourths read the guidance (see Figure 10). 


Acessibility Features and Accommodations 
Manual Items 
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80% 


69.90% 
70% 


60% 
m Computer-based 
Administration 
Aware n = 4,635 
Read n = 3,278 


50% 
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% Selected 'Yes' 


30% m@ Paper-based 
Administration 
Aware n = 1,687 


Read n = 1,104 


20% 


10% 


0% 


Aware Read (of those Aware) 


Figure 9. Percentage of TAs aware that there is an Accessibility Features and 


Accommodations Manual and, of those, percentage who read it. 

Percentages are based on the number of valid responses. Missing = 241 for CBT Aware and missing = 25 for PBT 
Aware; Missing = 248 for CBT Read and missing = 28 for PBT Read. CBT = Computer/Tablet-based administration; 
PBT = Paper-based administration. 
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Computer-Based Accessibility Features and Embedded 
Accommodations Available for PARCC Field Test Items 
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Figure 10. Percentage of TAs aware that there is approved PARCC guidance on computer- 
based accessibility features and embedded accommodations and, of those, percentage who 


read it. 
Percentages are based on the number of valid responses. Missing = 276 for Aware and missing = 292 for Read. 


The TA Surveys also included an open-ended question asking TAs to provide recommendations 
on how PARCC can improve the training and information available on accessibility features and 
accommodations. Several themes were identified, most of which dealt with recommendations 
for changing aspects of the overall system of accessibility features and accommodations, rather 
than changes to the actual training on accessibility features and accommodations. First, TAs 
recommended that all features and tools should be made available. These comments were 
made by TAs who indicated that specific accessibility features, such as text-to-speech, were 
sometimes not available to students, even though they stated that they had checked 
beforehand with Pearson or someone in charge of test coordination at the school to ensure 
that those features would be available during testing. Moreover, this theme included 
comments that features and tools that were available to students in the tutorials (e.g., 
magnifying glass, highlighter) were not available to them during testing. (It should be noted that 
not all accessibility features and accommodations were available at all grade levels during the 
field test; however, all approved accessibility features and accommodations are available for all 
grades for operational testing.) Second, TAs recommended that there should be a clear 
protocol for requesting accommodations for students. Relatedly, some TAs raised concerns 
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about potential legal ramifications if particular accommodations were not available for testing, 
but those accommodations are required by the students’ IEPs. Third, TAs commented that the 
amount of information provided in the instructions to students should be condensed. They felt 
that students were presented with too much extraneous information, especially for students 
with disabilities and/or ELL students. Fourth, TAs recommended that both students and TAs 
should have access to a full practice test prior to actual testing so that they can become familiar 
with aspects of the assessment such as, icons, where tools are located, and the general layout 
of the test. TAs felt that this would have helped them to be better prepared to answer student 
questions regarding the overall test and specific accessibility features and accommodations. 
Fifth, some TAs recommended more training, either because they did not receive the training 
on accessibility features and accommodations or because they felt that the training they did 
receive was insufficient to deal with problems that arose during testing. 


Finally, there appeared to be confusion with the read aloud accessibility 
feature/accommodation. Some TAs expressed confusion about the text-to-speech read aloud 
accessibility feature/accommodation versus the human reader read aloud accessibility 
feature/accommodation. The following comments from the TA Survey illustrate this confusion: 


e There is also confusion when to use human reader and when to have read aloud. We 
thought read aloud was built into the computer system and that clearly was not the case 
when we tried it (student pre-identified). 

e There is confusion as to reading to student: text to speech, and human _ reader, 
differences? 

e Instead of human reader is there a way the test could be read through the testing 
device? 

e acomputer-based assessment should not need a human reader! Please add a feature to 
the test where it can read the students the test questions/answers over the microphone. 


The first two comments above indicate confusion about when to use the text-to-speech 
accessibility feature/accommodation and when to use the human reader accessibility 
feature/accommodation, whereas the latter two comments indicate a lack of awareness of the 
text-to-speech accessibility feature/accommodation. 


Of the 957 TAs who provided recommendations on improving accessibility features and 
accommodations training (CBT administration), nearly half of those provided recommendations 
that dealt with one or more of the topics identified above. 


There was considerable overlap in the recommendations provided by TAs administering the 
PBT. The most common recommendations for improving accessibility features and 
accommodations training made by these TAs were to: (a) make all features and 
accommodations available during testing, (b) provide a clear protocol for requesting 
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accommodations for students, and (c) condense the instructions so that students with 
disabilities and ELL students do not have to process extraneous information. Of the 413 TAs 
who provided recommendations on improving accessibility features and accommodations 
training (PBT administration), nearly half provided recommendations that dealt with one or 
more of these three topics. 


The findings from the observation study reveal that the observers had little opportunity to 
observe separate accommodated sessions; only four of the 77 sessions (3 schools) included 
students receiving accommodations. For the task, “Ensure students receiving the same 
accommodations are grouped together,” two sessions (at 2 different schools) were rated as 
having “met” this criterion (see Table 6 above). The “not met” rating was used by an observer 
at another school (2 sessions) because the “General Administration Directions Clarified in 
Student’s Native Language” accommodation was administered with the regular test group 
instead of being in a separate testing location. Another separate accommodation session was 
scheduled to be observed; however, it was cancelled because the TA could not load Adobe 
Flash® on the computers. 


Additional information regarding accommodations was received during the debriefing 
interviews at seven schools. Comments from four schools indicated they had problems setting 
up the accommodations prior to testing, stating that the documentation to support this process 
was “insufficient.” It is important to note that they later discovered they had misunderstood 
the instructions. Three schools using the text-to-speech accessibility/accommodation indicated 
that they found it effective; the students were able to start, rewind, stop, and navigate through 
the test with very little support from the TA. However, there were two other schools that 
reported they were unable to get the text-to-speech accessibility/accommodation to work, and 
they abandoned their efforts to test students with IEPs. Finally, two schools said they revised 
IEPs to better match the accommodations provided on the PARCC assessment, and that they 
intend to apply that revision in their annual IEP reviews for the next year to better prepare for 
the assessment next spring. 


Overall, the findings related to accessibility features and accommodations suggest that efforts 
should be made to increase participation rates in training on accessibility features and 
accommodations—for both CBT and PBT administrations. Relatedly, efforts should be made to 
help ensure that all TAs are aware of and familiar with the PARCC manuals on accessibility 
features and accommodations. Particular attention should be given to addressing the concerns 
noted above with the read aloud accessibility feature/accommodation. Such efforts will help to 
tighten adherence to Standard 6.1. The findings also reveal that additional guidance and clarity 
are needed to the procedures for requesting and receiving accommodations; such efforts will 
help to strengthen adherence to Standard 6.2. To help address this particular concern, all 
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approved accessibility features and accommodations are available for all grades for operational 
testing. Other actions that PARCC is taking to address the concerns related to Research 
Question 2 are provided in the Executive Summary and in the Summary and Recommendations 
sections under Claims 1 and 2. 


Findings for Research Question 3: Do the students appear to understand the questions 
provided to them? 


The claim in the theory of action most directly related to Research Question 3 is, “Students 
respond to items as intended.” For this claim to be true, important assumptions that must be 
verified are that: (a) students understand the directions provided by the TA, and (b) students 
understand the directions on the test. Findings from the Student Surveys directly address these 
assumptions. ~ Upon completing the test, students were asked, “Did you understand all of the 
directions read by the person who gave you the test?” and “How often was it hard to 
understand the directions for the questions on the test?” The Student Survey results indicate 
that the vast majority of students indicated that they understood the directions read by the TA; 
this was somewhat more true for the ELA assessments than for the Mathematics assessments, 
and for the PBA assessments than for the EOY assessments, such that the lowest endorsement 
occurred for the Mathematics EOY assessments (regardless of CBT or PBT) with approximately 
78% indicating, ‘yes’ they understood all the directions read by the TA (see Figures 11 and 12). 


* Additional evidence for Research Question 3 is presented under Research Question 7b. Research Question 7b 
asks, “What types of questions, if any, do students ask during the test administration?” The TA Surveys included an 
open-ended question asking TAs to describe the types of questions asked by students. Many of the questions 
asked by students pertained to confusion about the instructions. Examples of those comments are provided in the 
results for Research Question 7b. 
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Performance-Based Assessment: 
Did you understand all of the directions read by the person who 
gave you the test? 
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Test Subject and Mode of Administration 


Figure 11. Percentage of students indicating they understood all the directions read by the TA 
(PBA). 

Percentages are based on the number of valid responses. ELA CBT missing = 170; Math CBT missing = 194; ELA PBT 
missing = 138; Math PBT missing = 253. ELA = English Language Arts; CBT = Computer/tablet-based Test; 

PBT = Paper-based Test. 
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End of Year Assessment: 
Did you understand all of the directions read by the person who 
gave you the test? 
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Figure 12. Percentage of students indicating they understood all the directions read by the TA 
(EOY). 

Percentages are based on the number of valid responses. ELA CBT missing = 90; Math CBT missing = 159; ELA PBT 
missing = 106; Math PBT missing = 176. ELA = English Language Arts; CBT = Computer/tablet-based Test; 

PBT = Paper-based Test. 


Next, with regard to students’ understanding of the directions for the questions on the test, 
approximately 20% of the students responding to the ELA Student Surveys indicated that it was 
hard to understand the directions on the test either “most of the time” or “almost all of the 
time” (see Figures 13 and 14). One-third of the students indicated that it was hard to 
understand the directions on the test “almost never.” These results were true for students 
responding to the ELA Student Surveys regardless of the assessment type (PBA or EOY) or mode 
of administration (CBT or PBT). For the Mathematics tests, results indicate that students had 
even more difficulty understanding the directions on the tests. A third or more of the students 
reported that it was hard to understand the directions on the test either “most of the time” or 
“almost all of the time” (see Figures 13 and 14). Students completing PBA Mathematics tests 
administered via computer or tablet reported the most difficulty in understanding directions on 
the test with 43% (see Figure 13) indicating that it was hard to understand the directions for 
questions on the test “most of the time” or “all of the time.” Only about 15% of students 
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responding to the Student Surveys for Mathematics indicated that it was hard to understand 
the directions for the questions on the test “almost never” (this was true regardless of the 
assessment type or mode of administration). 


Performance-based Assessment: 
How often was it hard to understand the directions for the 
questions on this test? 
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Figure 13. Percentage of students indicating that it was hard to understand the directions for 


the questions on the test most of the time or almost all of the time (PBA). 

Percentages are based on the number of valid responses. ELA CBT missing = 383; Math CBT missing = 347; ELA PBT 
missing = 170; Math PBT missing = 249. ELA = English Language Arts; CBT = Computer/tablet-based Test; 

PBT = Paper-based Test. Ratings made on a 4-point scale where 1 = Almost Always, 2 = Most of the Time, 3 = Some 
of the Time, 4 = Almost Never. Results reported here reflect the combined percentage of “Most of the Time” and 
“Almost Always.” 
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End of Year Assessment: 
How often was it hard to understand the directions for the 
questions on this test? 
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Figure 14. Percentage of students indicating that it was hard to understand the directions for 


the questions on the test most of the time or almost all of the time (EOY). 

Percentages are based on the number of valid responses. ELA CBT missing = 292; Math CBT missing = 359; ELA PBT 
missing = 131; Math PBT missing = 208. ELA = English Language Arts; CBT = Computer/tablet-based Test; 

PBT = Paper-based Test. Ratings made on a 4-point scale where 1 = Almost Always, 2 = Most of the Time, 3 = Some 
of the Time, 4 = Almost Never. Results reported here reflect the combined percentage of “Most of the Time” and 
“Almost Always.” 


The TA Surveys also addressed the question of whether students appeared to understand the 
instructions provided to them. The TAs were asked to indicate their agreement with the 
statements, “Students appeared to understand the instructions | read to them during test 
administration,” and “The instructions | read to the students covered all of the information 
necessary to take the test.” The results presented in Figure 15 indicate that approximately 75% 
of the TAs administering the CBTs agreed that students appeared to understand the 
instructions they read to them, whereas 50% of TAs administering PBTs agreed that students 
appeared to understand the instructions they read to them. The results presented in Figure 16 
indicate that about two-thirds of TAs administering CBTs agreed that the instructions they read 
to students covered all the information needed for students to take the test, and slightly more 
than half of the TAs administering the PBTs agreed with this statement. 
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Students appeared to understand the instructions | read to them 
during test administration. 
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Figure 15. Percentage of TAs agreeing that students appeared to understand the instructions 


they read to them. 

Percentages are based on the number of valid responses. Missing = 225 for CBT administration and missing = 29 
for PBT administration. Ratings made on a 4-point agreement scale where 1 = Strongly Disagree, 2 = Disagree, 

3 = Agree, and 4 = Strongly Agree. Results reported here reflect the combined percentage of “Agree” and “Strongly 
Agree.” 
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The instructions | read to the students covered all of the 
information necessary to take the test. 
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Figure 16. Percentage of TAs agreeing that the instructions they read to students covered all 


of the information students needed to take the test. 

Percentages are based on the number of valid responses. Missing = 244 for CBT administration and missing = 26 
for PBT administration. Ratings made on a 4-point agreement scale where 1 = Strongly Disagree, 2 = Disagree, 

3 = Agree, and 4 = Strongly Agree. Results reported here reflect the combined percentage of “Agree” and “Strongly 
Agree.” 


A factor that may affect whether students understood the instructions provided to them is 
whether they had the opportunity to gain familiarity with the new PARCC assessments prior to 
actual administration. Given that the PARCC assessments are brand new to students, it is 
reasonable to expect that students may be better positioned to understand the instructions if 
they had already practiced with PARCC sample items. Practice with sample items is consistent 
with the guidance for Standards 4.16 and 6.5. The TA Surveys helped to investigate this 
question by asking TAs to indicate, “Did the students in your session(s) practice with PARCC 
sample items prior to administration?” and “Did the students in your session(s) complete the 
PARCC tutorial prior to administration 
fourths of TAs administering the CBT indicated that students practiced with sample items prior 
to administration, whereas only about a fourth of TAs administering the PBT indicated that 
students practiced with sample items prior to administration. Relatedly, approximately three- 


?”?° The results in Figure 17 reveal that more than three- 


?° This information also directly addresses the Implementation Phase in the TOA, most notably the claim that 
“Students understand the format of the assessments and how to use the technology.” 
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fourths of TAs administering the CBT indicated that students completed the PARCC tutorial 
prior to administration, and less than a tenth of TAs administering the PBT indicated that 
students completed the PARCC tutorial prior to administration (see Figure 18). With regard to 
the results for PBT administration, it should be noted that were no sample items or tutorial 
specific to PBT; this explains why percentages for PBT administration in Figures 17 and 18 are 
low. Even though the majority of TAs indicated that students taking the CBTs had gained 
familiarity with the PARCC assessments prior to administration via sample items and/or via the 
PARCC tutorial, the survey results reveal that a substantive number of students did not 
complete either of these; this might partly explain why there was some confusion with 
instructions on the test. 


Did students practice with PARCC sample items prior to 
adminstration? 
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Figure 17. Percentage of TAs indicating that students practiced with PARCC sample items 


prior to administration. 
Percentages are based on the number of valid responses. Missing = 389 for CBT administration and missing = 80 
for PBT administration. 
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Did students complete the PARCC tutorial prior to administration? 
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Figure 18. Percentage of TAs indicating that students completed the PARCC tutorial prior to 


administration. 
Percentages are based on the number of valid responses. Missing = 387 for CBT administration and missing = 76 
for PBT administration. 


Finally, findings from the observation study also shed some light on Research Question 3 by 
providing concrete examples of the kinds of confusion encountered by students with regard to 
instructions. For example, as initially discussed in the results for Research Question 1, observer 
notes from six schools indicate that confusion was caused by terminology in the scripts. For 
example, observers noted the following: (a) the directions instructed students to click the ‘start’ 
button, but the button is called ‘resume,’ (b) the TA had to clarify that the ‘testing device’ is the 
computer, and (c) that “sections” and “sessions” mean different things in different places. 
Furthermore, observer notes from five schools indicated that students spent considerable time 
figuring out how to undo what a tool did (e.g., highlighting tools). Most of the notes regarding 
tool usage were from the EOY test administration. Several observer notes were specific to the 
Mathematics tests (CBT administration). For example, observers noted that students were 
confused as to how to enter fractions into the equation editor and how to change answers 
when plotting points in a graph. Observers also noted that the answer boxes did not allow 
commas (e.g., to indicate thousands) so students had to go back and delete commas before 
submitting their answers on the Mathematics CBTs. (See the results for Research Question 7b 
for more information on the types of questions students asked TAs about test instructions). 


These concrete examples from the observations help shed some light on findings from the 
Student Surveys and findings from the TA Surveys regarding confusion in instructions, 
particularly for the Mathematics CBTs for which 43% of students reported that the instructions 
were hard to understand all or most of the time. 
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Finally, an additional theme surfaced in the debriefing interviews regarding sources of 
confusion with instructions. Five schools expressed concern about younger and “at-risk” 
students having adequate keyboarding or computer experience to reasonably respond to 
various types of answer options and multistep test items. This concern was supported by 
observations of some students who did not know simple keyboarding skills, such as how to use 
the scroll bar. 


In summary, most students reported that they understood the directions read to them by the 
TA. The majority of TAs also reported that students appeared to understand the directions they 
read to them; however, agreement with this perception was considerably less for TAs than for 
students, particularly for TAs administering PBTs. Furthermore, only about half to two-thirds of 
the TAs agreed that the instructions they read to students covered all of the information 
students needed to take the test. With regard to the instructions students read on the test, 
students reported more difficulty in understanding those directions than in understanding the 
directions read to them by the TAs; this was particularly true for mathematics. Per Standards 
4.16 and 6.5, students are more likely to understand test instructions if they have had practice 
with the items and with the mode of testing (e.g., computer-administered) prior to actual 
testing. Overall, the findings indicate that approximately three-fourths of students taking the 
CBTs completed practice items and the tutorial prior to testing. While some students who took 
the PBTs completed the practice items and/or tutorial, those activities were targeted for CBTs, 
not PBTs. There were no specific practice items or tutorials for PBTs. This might partly explain 
why TAs administering PBTs generally reported lower levels of agreement that students 
understood the instructions they read to them. Additional clarity and detail in test instructions 
and increased practice opportunities are two efforts that could help tighten adherence to 
Standards 4.16 and 6.5. Actions that PARCC is taking to address these concerns are provided in 
the Executive Summary and in the Summary and Recommendations sections under Claim 4. 


Findings for Research Question 4: To what degree are students engaged in taking the test? 


Research Question 4 also most directly addresses the claim that, “Students respond to items as 
intended.” For this claim to be met, students must have some engagement with the 
assessment. Consequently, another assumption underlying the claim “that students respond to 
items as intended” is that students are engaged in taking the test. If students are not engaged 
or if they have low levels of engagement, then they might not sufficiently attend to the items so 
as to respond to them as intended. 


Across the various studies and instruments, several sources of information were collected that 
help to shed light on the issue of student engagement. First, the observer ratings from the 
observation study provide an indication of the extent to which certain kinds of potential 
distractors were minimized. For example, ratings from the observation checklist indicate that, 
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of the 77 observed sessions, more than 98%” were rated as having ‘Met’ the task of: “Allow 
authorized visitors only.” This indicates that student engagement was not likely to have been 
impeded by outside visitors. 


Qualitative evidence of student engagement comes from two additional sources—students’ 
open-ended comments on the Student Surveys and TAs’ comments from the debriefing 
interviews.” The Student Surveys concluded with an open-ended question asking students, 
“Please use the space below to tell us about anything that was confusing or unclear about the 
test. We are not asking whether a question on the test was hard, but we would like to know if 
there was anything about the test that didn’t make sense.” Even though this open-ended 
question was directed at identifying confusing content, a substantive number of students (i.e., 
approximately 15% of all students providing open-ended comments) took the opportunity to 
report that they “liked” the test or that they found it “interesting,” “fun,” or “cool.” These kinds 
of comments suggest engagement, at least for some students. Relevant examples of student 
comments are: 


e Nolactually really liked taking this test (: only issue would be the speed of the PCs or 
tablets , my tablet seemed to work fine then it would take its time like (type slowly) . 
Otherwise its a really cool/ interactive way to test. [PBA Math Grade 8] 

e |thoght this test was so awesome. Nothing was confusing. | liked the test on the 
computer alot better than on paper. | understood everything my hands weren't getting 
tired from writing because | was on the computer typing. | dont think you could've made 


e test was cool because i have a chance to do more test in the comuter it was cool becuse 
it is very osom to do a test on acoputer and i will like to do more test in the coputer and 
some of the questions were dificlt [PBA ELA Grade 5] 

e Onmy test there was nothing that didnt make sense. Actauly my test was fun a littel bit 
hard at the same time. Next year | hope it will be the same way. [EOY ELA Grade 3] 


Converging evidence of student engagement was obtained from the debriefing interviews. 
While the debriefing interview protocol did not include a specific interview question regarding 
student engagement, during the interviews fourteen schools commented that students liked 
the CBTs overall with typical comments such as, “students stated they preferred taking an 
assessment online compared to paper and penci 


|,” M“ 
, 


students thought the videos were cool,” 
“kids enjoyed the test format...loved the cut and paste function as a way of citing evidence,” 
and “students seemed engaged and mentioned they enjoyed the format of many items.” 


77 All “not applicable” testing sessions have been removed from the percentage reported here. 

* The qualitative evidence of student engagement from the Student Survey and from the debriefing interview is 
only pertinent to CBTs; the Student Survey administered to students taking the PBTs did not contain an open- 
ended item and the observation study only included schools administering CBTs. 
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Additionally, some TAs described characteristics of the items that engaged students. For 
example, the on-screen math tools and video segments were mentioned during several 
interviews as positive features of the assessment. Comments from six schools indicated that 
items were either more difficult or required more rigor than their current accountability 
assessments. One of the six schools indicated they will make recommendations to their building 
and district administration to better align instructional practice to the rigor of these 
assessments. 


There was also some evidence contrary to the evidence provided above. There is converging 
evidence across the various studies and instruments that problems related to technology were 
a likely impediment to student engagement. Findings from the TA Survey indicate that only 13% 
of TAs reported that they encountered “no technology-related problems” during test 
administration. The students appeared to be less aware of technology-related problems than 
the TAs; nonetheless, the percentage of students indicating that there were no technology- 
related problems during the test was between 40% - 60% (see Figure 19). Relatedly, observer 
notes from the observation study indicate that there were technology issues (e.g., not being 
able to sign in, timing out, kicked-off) at 49 of the 61 schools visited (80%). The problems with 
technology were identified as either “severe” in nature, defined as an interruption in test-taking 
of at least 10 minutes, or as “minor” in nature, defined as problems that were resolved in less 
than 10 minutes. During field test observations, 40 schools experienced an issue(s) categorized 
as minor, and 20 schools experienced an issue(s) categorized as severe(some schools 
encountered both minor and severe problems). More detail on the observer notes from those 
sessions is provided below in the results for Research Question 7a (Are there any hardware or 
software related problems during test administration?). 
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No Technology-Related Problems Occurred (Students) 
90% 


80% 
70% 
60% 


50% 


39.50% 


40% 


% Indicating No Problems 


30% 


20% 


10% 


0% 
PBA ELA PBA Math EOY ELA EOY Math 
n= 221,158 n= 183,099 n = 160,820 n = 157,872 


Figure 19. Percentage of students indicating there were no problems related to technology 


during their test. 
PBA = Performance-based Assessment; EOY = End-of-Year; ELA = English Language Arts. 


As a consequence of the technology problems, students reported that in some instances they 
skipped items that were not functioning correctly or that they rushed to finish the test because 
of the delays caused by technology-related problems. For example, comments made by 
students on the Student Surveys included: 


e The internet was not responding very well and | sometimes couldn’t sign in to take my 
test. Most of the time when! try to sign in, a pop up will come and it’s an EO. Fixing the 
computer would take so much time that sometimes | had to rush my test once | was able 
to sign in. [PBA ELA Grade 8] 


Updated February 17, 2015 Page 49 


PARCC Quality of Test Administration 


e |hadan extremely difficult time getting to the test because of either problems with the 
test itself or the computer, this caused me to have less than half of my time to finish, 
which made me feel very rushed. [PBA ELA Grade 9] 

e There was a lot of difficulty when attempting to select an answer and move it to the 
correct box, scores will be a lot lower because people had to skip over questions that did 
not work. [EOY ELA Grade 10] 

e Onone question of the test, we had to drag evidence to support our answer out of the 
selection. | was not able to get it to work so | skipped the question; all | could do was 
highlight it [EOY ELA Grade 6] 

e On the test part my digital keyboard did not work. On paper this test would of took me 
30 minutes, but on the computer it took me 2 hours. When | hit next question, it took 
about thirty seconds to load so | felt rushed when time was running short. [EOY Math 
Grade 4] 


If students are rushing through the test to finish on time and/or skipping items, then student 
engagement is negatively impacted, which means students might not be responding to items as 
intended. It is unknown whether students would have been more engaged, irrespective of 
technology issues, if the test had been operational rather than field test and/or if student-level 
stakes were attached to scores. 


Additional themes regarding problems with technology were identified in the open-ended 
comments on both the TA Survey and the Student Surveys, and during the debriefing interview 
with TAs; more detail is provided below in the results for Research Question 7a. 


Another potential impediment to student engagement is the lengthiness of the instructions 
read by the TAs. The lengthiness of the TA script was connected with contributing to students’ 
lack of attention. For example, typical notes made by four observers during the observation 
study were: “Students quit listening after page 31,” “...so redundant (for a consecutive 


Dl 


session),” “...kids not paying attention,” and “LOTS of instructions to listen to (took 15 minutes 
to read).” Evidence from the TA Surveys is consistent with the reports from the observation 
study. The TA Surveys included an open-ended item asking TAs to, “describe any questions 
students asked about the test-taking instructions you read to them.” A prevalent theme 
identified in these open-ended comments was that the instructions were too long. A typical 
comment from TAs was, “Directions were too long—students either stopped listening or were 
unable to retain that much information at one time.” The issue of retention/comprehension of 
the lengthy instructions was noted as a particular concern for students with an IEP and/or ELL 
students. For example, a typical comment made by TAs was, “I found the lengthy directions 


were way too hard for students to follow. We have a high number of ELL and IEP students and 
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the amount of information given allowed no processing time for the typical students let alone 
our ELL and IEP students.” 


In summary, there is some evidence of student engagement that comes from both the Student 
Surveys and the observation study. Qualitative comments from some students on the Student 
Surveys indicate that they “liked” the CBT. Also, comments made by TAs during the debriefing 
interviews indicate some reports of students preferring the features of the CBT (note that 
observers only visited CBT administrations). Moreover, findings from the observation study 
indicate that student engagement was not likely to have been impeded by unauthorized visitors 
in the testing session. Diverging evidence of student engagement, however, comes from 
reports of technological-related issues. Evidence of technology-related problems comes from 
the TA Survey, the Student Surveys, and the observation study. Qualitative comments on the 
Student Surveys indicate that some students skipped items or rushed through the test in order 
to make up for delays caused by technology-related problems. Efforts to rectify technological 
issues should help to minimize disruptions and thereby increase adherence to Standards 6.3 
and 6.4. Also, evidence from the observation study and the TA Surveys suggest that the 
instructions read by the TAs were too lengthy and contained redundant information 
(particularly for students participating in consecutive testing sessions), which contributed to 
lapses in attention for test-takers. Consequently, efforts to streamline the instructions read by 
the TAs and to remove and/or reduce redundancies in instructions could help to improve 
student attention and engagement, and thereby increase adherence to Standards 4.16 and 6.5. 
Actions that PARCC is taking to address these concerns are provided in the Executive Summary 
and in the Summary and Recommendations sections under Claim 4. 


Findings for Research Question 5: Is there any disruptive student behavior during the session? 


This research question also gets to the heart of the claim that “students respond to items as 
intended.” Another assumption underlying this claim is that students’ attention is not adversely 
impacted by other disruptive students. To the extent that disruptive student behavior is 
minimized, then this increases the likelihood that students attend to the test and respond to 
items as intended. 


Across the various studies and instruments, several sources of information were collected that 
help to shed light on this research question. First, the observer ratings from the observation 
study provide an indication of the extent to which disruptive student behavior was minimized. 
Ratings from the observation checklist indicate that, of the 77 observed sessions, more than 
90%* were rated as having ‘Met’ the tasks of: “Seat students at a computer with non- 
permitted materials removed (i.e., cell phones, other electronics, and reference books),” 


3 All “not applicable” testing sessions have been removed from the percentage reported here. 
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“Monitor students by circulating around the room,” “Handle test irregularities (e.g., student 
misconduct, cheating, using electronic devices),” and “Handle students finishing early 
appropriately (i.e., following state guidance).” This information helps support the assumption 
that students’ attention was not adversely impacted by disruptive student behavior. 


While disruptive student behavior does not appear to be a widespread concern, there was 
some evidence of instances of disruptive student behavior across studies. First, from the 
observation study there were reports of disruptive student behavior in two sessions (at two 
different schools). The observer notes from one school indicate that, “... (there was) 
disrespectful talking, students being physical with each other. There were some attempts to 
control the misbehavior.” The observer notes from the second school indicate that, “student 
arrived late; was unwilling to write his name on scratch paper; then refused to answer any 
questions. The test coordinator removed him from the room and brought him back later 
because he had agreed to participate. He logged in while the other students waited to be read 
item #5 then immediately hit the Enter key to go through the assessment and the Student 
Survey without responding.” 


From the Student Surveys, there were some reports of disruptive student behavior, but it was 
rare. Some example comments provided by students were: 


e Sometimes, my classmates would become disruptive, and they would make me lose 
focus. 

e When we were done we had nothing to do so if we did anything we would be distracting 
other people. | did not like that part. 

e |couldn't really focus and concentrate as much because of the test coordinator 
continually screaming and the classmates that were in the room with me kept talking 


The open-ended comments provided by TAs on the TA Surveys help to provide some additional 
context to the student comments listed above. The TAs noted that students who finished early 
were often times “fidgety,” and thus distracting to other students still taking the test. These TAs 
commonly mentioned that students should be allowed to read a book as soon as they finish 
their test to help stave off such “fidgety” behavior. Example comments from TAs were: 


e The fact that students need to wait until the end of the initial testing session before 
being allowed reading materials did lead to more kids beginning to get fidgety. 

e Students need to be able to turn in their tests when they are done and be able to read a 
book immediately. Watching 20 kids fidget for 15-20 minutes is distracting for those still 
working. 

e Although the children could read a recreational book when finished, the directions 
seemed to indicate that they could not do this until the additional time period, which 
meant that they were sitting there doing nothing after they checked their test. This is a 
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waste of time and can lead to potential behavior problems. If this is not right, the 
directions need to be clarified. 


In summary, the findings for Research Question 5 indicate that disruptions caused by students’ 
behavior during the testing session were not widespread. Given that such disruptions were very 
limited, there are no planned PARCC actions in the Lessons Learned document to address 
disruptive student behavior. The most common reports of disruptive student behavior were 
reports of students being “fidgety” after finishing the test early. The most common 
recommendation made by TAs was to allow students to read a book as soon as they completed 
their test to help reduce such fidgety behavior. ~* This might be one way of helping to further 
increase adherence to Standard 6.4. 


Findings for Research Question 6: Are there any apparent attempts to record or copy test 
materials including the test questions by students or others? 


This research question most directly addresses the claim that, “Administrators and teachers are 
prepared to administer the assessments as intended.” An important component of test 
administration is ensuring that the test content remains secure. To do otherwise would 
threaten the validity of the testing system. A source of evidence for this research question is 
the observation study. Several tasks on the observation checklist pertain to issues related to 
test security. That collective set of selected tasks and the degree to which they were met (as 
determined by the trained observers) are presented in Table 8. The results in Table 8 indicate 
that most of the tasks related to test security were rated as “Met.” Tasks that did not have a 
high percentage of “Met” ratings were typically because that task was either “Not Observed” or 
was “Not Applicable.” Notable exceptions to the “Met” ratings occurred for “prepare testing 
room” and “post testing signs on outside doors.” Observer notes from 20 schools (26 
observations) indicate that the primary issue with room preparation was with the lack of use of 
partitions or lack of sufficient space between computers as required. Typical comments were: 
“not enough seats (space) to put extra space” and “students were close together but 
accustomed to it.” With regard to posting of the “Testing — Do Not Disturb” sign, there were no 
observer notes of significance to indicate anything other than TAs neglecting to post the sign 
(such as significant interruptions caused by failure to post the sign). It is important to note that 
a “Not Observed” rating only indicates that the observer did not see the task completed; 
therefore, as noted above, in many cases the task was completed. 


One observer gave a rating of “did not meet” for the “Allow authorized visitors only” task with a 
note that an ELA teacher entered the room and stayed about 2 minutes after looking at a 
couple of screens. The observer noted that the TA and the district assessment director did not 


** Decisions about what students are allowed to do after they complete testing is a state policy issue. 
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take any overt action at that time. For the task of “Maintain test security if test is interrupted,” 
one observer provided a “did not meet” rating because the TAs (multiple classrooms testing 
concurrently) sent students back and forth between the room with the Test Coordinator and 
the room with the administration computer to try to get reset. 


Table 8. Summary of Observation Checklist Items Pertinent to Test Security 


Did 
Partially Not Not 
Met Met Meet Observed NA 
% % % % % 

Prepare testing room (every other seat or 63.6 15.6 18.2 13 13 
partitioned, room configured for security/privacy). 
Post "Testing - Do Not Disturb" sign on outside 51.9 3.9 21 208 13 
door(s). 
Confirm correct Student Authorization Tickets and 30.5 0.0 0.0 18.2 13 
seal codes for session. 
Allow authorized visitors only, such as PARCC staff, 33.1 0.0 13 0.0 15.6 
state/local/school assessment monitors. 
Seat students at a computer. Materials not allowed 
are cell phones, other electronics, or reference 96.1 2.6 0.0 1.3 0.0 
books. 
Provide students with a Student Authorization 97.4 0.0 0.0 13 13 
Ticket. 
Monitor students by circulating around the room. 97.4 2.6 0.0 0.0 0.0 
Provide a group 3 minute break if necessary 
(computer screens must be turned off or apply 
visual block to monitor). Students must not noe oe a oe ane 
communicate during breaks. 
Maintain test security if test is interrupted. 6.5 0.0 1.3 0.0 92.2 
Handle testing irregularities (e.g., student 
misconduct, cheating, using electronic devices, fire 23.4 1.3 0.0 0.0 75.3 
drills or safety threat, loss of internet connectivity). 
Collect Student Authorization Ticket and any test 38.3 39 00 26 52 


materials (including all scratch paper). 


Return testing materials to the Test Coordinator. 48.1 0.0 0.0 23.4 28.5 


Note. NA= Not applicable. 
Number of observed testing sessions is n = 77. 


In summary, there was no evidence of overt attempts to record or copy test materials by 
students or others. Findings from the observation study indicate that most of the tasks related 
to test security were rated as “Met,” thereby indicating evidence to support adherence to 
Standard 6.7. Exceptions to the “Met” ratings occurred for “prepare testing room” and “post 
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testing signs on outside doors.” Observer notes indicate that the primary issue with room 
preparation was with the lack of use of partitions and the lack of sufficient space between 
computers. With regard to posting of the “Testing — Do Not Disturb” sign, there were no 
observer notes of significance to indicate anything other than TAs neglecting to post the sign. 
Given that there was only one reported instance of an unauthorized visitor entering a testing 
session, not posting the “Do Not Disturb” sign does not appear to be a major threat to test 
security. Moreover, some schools have an “open concept” in which there are no doors to 
classrooms and/or classrooms are joined together. Consequently, posting the “Do Not Disturb” 
sign might not be practically feasible in some settings. Therefore, based on findings from the 
field test, it appears that addressing the lack of sufficient space and/or lack of partitions 
between computers is the test preparation task that is in most need of improvement. Increased 
following of this test administration instruction will help to tighten adherence to Standard 6.6. 


Findings for Research Question 7a: Are there any interruptions during the test administration 
related to hardware or software problems? 


This research question is most closely related to the claim that, “Technology improves and 
facilitates the assessment experience.” If there are interruptions caused by problems with 
hardware and/or software, then the veracity of this claim is threatened. Both the TA Survey and 
the Student Surveys (CBT versions) include questions that address this research question. The 
TA Survey asked TAs to select which, if any, of the following types of technology-related 
problems occurred during test administration: difficulty logging into the assessment, devices 
stopped working, devices worked slowly, and lost internet connection. As reported earlier, 
under Research Question 4, only 13% of TAs indicated that no technology-related problems 
occurred during test administration. The results for the types of technology-related problems 
encountered by the remaining 87% are presented in Figure 20. These findings indicate that 
technology-related problems were not uncommon, and that the most frequently encountered 
problems were with logging onto the assessment and with devices locking up (i.e., “stopped 
working”). 
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Of those indicating they encountered technology-related problems, which 
kind did they encounter? (select all that apply). (TAs) 


52.60% 


33.30% 


% Selected Problem Occurred 
uw 
oO 
x 


Difficulty logging on Devices stopped working Devices worked slowly Lost internet connection 


Computer/tablet-based Administration 
n = 4046 


Figure 20. Percentage of TAs indicating particular kinds of technology-related problems 


occurred during test administration. 
TAs indicating no technology-related problems were excluded from this analysis. 


As reported under Research Question 4, the percentage of students indicating that there were 
no technology-related problems during the test was between 40% - 60%. The Student Surveys 
also asked students to indicate which kinds of technology problems occurred while they were 
taking the test. The types of technology-related problems indicated by the approximately half 
that did indicate problems are presented in Figure 21 (for PBA) and Figure 22 (for EOY). The 
results in those figures indicate that, overall, more technology-related problems were reported 
during PBA administration than during EOY administration. An exception to this pattern 
occurred for “Difficulty dragging or moving objects on screen,” such that more students 
reported this being a problem on the EOY assessment than on the PBA. The most common 
technological problem—difficulty making changes to answers—was reported for students 
taking the Mathematics PBAs on the computer/tablet, with two-thirds of students indicating 
that this problem occurred. Other notable findings displayed in Figures 21 and 22 are that more 
than half of the students said their computer worked slowly on the PBA, and slightly less than 
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half said their computer worked slowly on the EOY assessment. In addition, more than a third 
reported that their computer stopped working on the ELA PBA. 


% Selected 


Of those indicating they encountered technology-related problems, which 
kind did they encounter? (Select all that apply). (Students) 


100% 


90% 


80% 


70% 


60% 


@ ELA CBT 
n= 221,158 


=m Math CBT 
n= 183,099 


50% 


40% 


30% 


20% 


10% 


0% 


Computer Stopped Computer Worked __ Difficulty Dragging or Difficulty Making 
Working Slowly Moving Objects on Changes to Answers 
Screen 


Figure 21. Percentage of students indicating particular kinds of technology-related problems 


occurred during test administration (PBA). 
Individuals who selected ‘No problems’ were excluded from this analysis. ELA = English Language Arts; 
CBT = Computer/tablet-based Test. 
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End of Year Assessment: 
Did any of the problems listed below happen during your test? 
(Select all that apply). 
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Figure 22. Percentage of students indicating particular kinds of technology-related problems 


occurred during test administration (EOY). 
Individuals who selected ‘No problems’ were excluded from this analysis. ELA = English Language Arts; 
CBT = Computer/tablet-based Test. 


While the survey findings suggest that students might not have been as aware of technology- 
related problems as the TAs, the results from both the TA Survey and the Student Surveys 
indicate the presence of some technology-related problems. The findings from the observation 
study provide converging evidence to support this finding, and also provide an indication of the 
degree of the problem. Observer notes indicate that there were technology problems (e.g., not 
being able to sign in, timing out, kicked-off) in 52 observed sessions at 49 of the 61 schools 
visited (80%). The technology issues were categorized as severe or minor. Severe technology 
issues were defined as those that interrupted testing for 10 minutes or more, while minor 
issues were defined as those that were resolved in less than 10 minutes. Some schools 
experienced multiple technology problems, some minor and some severe. 


Tables 9 and 10 provide examples of the observer notes for each of the types of technology 
issues identified by the observers. Table 9 indicates that, of the 20 schools (in 20 sessions) 
encountering severe issues, the majority of those were caused by problems with signing in and 
with the assessment shutting down/kicking students out. Seven of those schools experienced 
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delays that took more than an hour to resolve, and in five of the seven schools testing was not 


completed for all students that day. All seven of these schools were part of the PBA test 


administration. There were five EOY schools in the severe category; however, all instances were 


resolved in less than 30 minutes. 


Table 9. Summary of Observation Checklist Notes: Severe Technology Issues 


Technology Issues 


Testing suspended or 
incomplete 


# of Schools 


Example Observer Notes 
e All but three kids are nearing finishing the first session 
(1300) they had started at 0900 and had so many 
problems... 
e not really much observing occurring...so many problems 
getting students logged in and back in after their screens 
freeze up. The accommodation session was cancelled... 


Sign in 


12 


e Popup blockers have to be turned off they kept kicking 
students out. Now they got the orange box to be logged 
in by proctor...none of this in script... 

e 28 minutes...the delay was because they needed to add 
one student to the roster...they are experienced at using 
the TA management tool but it still took quite a bit of 
time. 

e ...the rest (of students) began 40 minutes after the 
period started because of all the login difficulties. 


Loading/timed out 


e Various tech problems interrupted the test throughout 
so it is impossible to know whether students would have 
requested extra time in a normal session. 

e Some of the early test items took a long time to load...in 
the end only 2 students (out of 18 regular 
students)...finished the test. 


Shutdown, kicked-off 


11 


e one student was kicked off and says her test is 
completed. They have tried to get her back in and the 
system won't allow it so they sent the student to class. 

e Locking up at "loading" in between questions.- When 
answering questions keyboard is lagging/freezing 

e Two students were bounced out of the system right after 
login...After a long delay they both got an erroneous 
error that Java was not installed. 


Note. Twenty schools experienced “severe” technology problems. Some of those schools experienced multiple 
issues; therefore the sum of the number of schools in the table is greater than 20. 


Table 10 shows that, of the 40 schools (in 43 sessions) identified as having minor technology 


issues, the majority of issues were also caused by problems with signing in and with the 


assessment shutting down/kicking students out. Minor technology issues were experienced at 


similar frequencies for PBA and EOY administrations. Seven schools had sign in problems due to 
the operating systems, Java® updates, and controlling pop-ups. Technical staff from one county 
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school system developed a shortcut to TestNav that also disabled popup blockers. Another 


school system’s technical team suggested that PARCC should develop an application that 


essentially accomplishes the same outcome—that is, disable popup blockers and access 


TestNav, regardless of the equipment, operating system, or Java® version. 


Table 10. Summary of Observation Checklist Notes: Minor Technology Issues 


Technology Issues 


Sign in 


# of Schools 


20 


Example Observer Notes 
The TA moved the student to another computer and he 
was able to log in and complete the survey. 
one student received an error message that said the Seal 
Code was incorrect but the student was already inside the 
cognitive portion of the test...logged out then back in. 
Needed to navigate through java and pop-up blocker 
screens. 
Effort to "resume" tests failed because all students to 
include those in "ready" status were checked. The error 
was corrected within one minute and all students were 
able to start the test. 


Loading/timed out 


Shutdown, kicked-off 


25 


When student computers time out their screens show 
they are logged off. But the admin computer shows they 
are still active. 

Test-timed out during intro when teacher was reading the 
script. 

Some students are seeing a strange character on their 
monitor...character is a dark triangle with a white question 
mark in it. 

Four students were kicked off when they started the test 
and had to log back in. 

A few kids had to be reset didn't take but 5 minutes to get 
them all started. One student got an error message, 
logged him out and back in. 

One student's computer locked up during the student 
survey. Authorization tickets had already been collected 
and his ticket had to be located so he could log back in 
Two students received Error 3005 at different points in the 
assessment - so another program was running in the 
background 


Note. Forty schools experienced “minor” technology problems. Some of those schools experienced multiple issues; 
therefore the sum of the number of schools in the table is greater than 40. 


Comments received from the debriefing interviews are consistent with observer notes 


describing technology issues. Generally, the number and severity of problems described by TAs 


were less frequent during the EOY administrations than the PBA administrations. In fact, one 


additional question asked during the EOY debriefing interviews was to compare technology 


issues to what TAs experienced during PBA administrations. Seven schools indicated they 
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participated in both test administrations and all agreed their EOY experience was better than 
their PBA experience with regard to technology-related issues. 


Similarly, technology-related problems were a common theme in responses to the open-ended 
items on the TA Survey and the Student Surveys. The most commonly mentioned problems 
were with log-on”’ and log-off and with saving/recovering student test data, which was a 
consequence of devices locking up and getting kicked off the system. 


Some illustrative comments provided by TAs on the TA Survey of technology-related problems 
26 
were: 


e Students frequently got stuck on pages that would not advance or would not auto-save. 
Each time we were stuck we had to re-login and re-fresh the student page. When this 
occurred on an open response page sometimes a student had to re-type lost information. 
Java was a bit of a pain, we had to re-install that on some computers daily--I don't know 
if this had to do with the computer or the program but it was a bother. 

e TestNav needs to run faster and allow more students to be logging on at the same time. 

e There were several glitches in the system that seem to need to be worked out before the 
test can run smoothly. Some students were logged out for no reason on day two, as well 
as slow computers and freeze ups when typing extended responses. 

e Students repeatedly had issues where pages loaded slowly or froze, not allowing them to 
continue their train of thought. Also they had issues with their work being deleted and 
not being able to recover it. 


Some illustrative comments provided by students on the Student Surveys of technology-related 
problems were:”” 


e |havea problem when the keyboard does not let me use the caps lock when i need to 
use it and when | tried to type something it makes a spelling error like this ( like changed 
to ike). Last time when | was doing a response it logged me out then | went back it was 
okay, but before that happen | ws typing my writing about reading on this and it deleted 
it. Was turned into ws and about turned into abot. | think there should be some fixes on 
the keyboard and testnav. there was some errors too so please fix testnav it would great 
with none of this happening [ELA PBA Grade 6] 

e That thing | did not like about this test is that the internet was very slow and that it took 
at least an hour and a half to log in and take the test. [ELA PBA Grade 8] 


*° The term “log on” and “sign in” were used interchangeably. 

°° See page 55 for discussion of the percentage of TAs encountering technology-related problems, and Figure 20 
for the percentages of TAs indicating that they encountered specific types of technology-related problems. 

*” See page 56 for discussion of the percentage of students encountering technology-related problems, and Figures 
21 and 22 for the percentages of students encountering specific types of technology-related problems. Also, see 
Figures 23 -30. 
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e Like every time | tried to log on my computer froze, and it did not let me move on to the 
next section. Even when | put the right code in. When | tried to click an answer it 
wouldnt let me, and it made the computer restart, and | had to do the test all over 
again. [ELA EOY Grade 6] 

e 1|did not like this test. It has too many issues with logging in and highlighting. | would not 
like to take this in the future due to all the issues. It is easier to focus better with words 
on paper. You get easily booted off of the site and it is very time consuming. [ELA EOY 
Grade 9] 

e It took a bit of time to log in; there were some errors that prevented us from logging in. 
[Math PBA Grade 5] 

e Halfway through section 1, my keyboard stopped working and | had to type my answers 
using the iPad keyboard. When | tried to delete a letter or unneeded space in my answer, 
the entire answer got deleted. Sometimes the iPad would freeze when | tried to scroll 
down. [Math PBA Grade 8] 

e There were a ton of problems with students trying to log on and it took way over an hour 
to figure out why. [Math EOY Algebra 1] 

e When! was trying to type my answer it would delete my answer and when ! had went to 
log out and | needed to log back in it would give me a notification saying | don’t have 
authorization. [Math EOY Grade 7] 


Another technology-related topic identified from the observer notes involves preparing the 
computer for test administration. A summary of topics on computer preparation is provided in 
Table 11. An issue identified from the debriefing interviews as common to four schools 
administering PBAs and three schools administering EOY assessments was learning through trial 
and error that the audio needed to be unmuted and volume adjusted before logging into 
TestNav. If not, students had to sign out of TestNav to make the adjustment then sign in again 
to resume testing. Observer notes from two schools, in Table 11, indicated technology staff 
created shortcuts to help speed up the computer preparation. Observer notes also indicated 
that schools varied widely in their experience with operating system software. Four schools 
stopped using Internet Explorer, and experienced fewer problems with Firefox and Google 
Chrome. Six schools described Java® pop-ups interfering with the assessment, in some cases 
immediately after the PARCC SystemCheck tool indicated that their computers were set up 
correctly. 
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Table 11. Summary of Observation Checklist Notes: Computer Preparation 


Computer Preparation 


We) o) (3 te) mY ot Tele) [5 Example Observer Notes 


e Students had to unmute audio prior to logging on to the 
test or else they would have to log off and back on. 


e Atthis school they'd set up icons with the desktop 
rather than having kids navigate to the TestNav. 

e Someone at the County Board of Education created a 
shortcut to TestNav for all schools. The shortcut 

5 disabled popup blockers and brought up the TestNav 

welcome screen. This allowed for quick and efficient 
preparation of the computers. 

e The tech had to disable the antivirus and the windows 
popups so they have to touch every computer. 


Headphones and volume 4 


Operating system and 
shortcuts 


The final issue related to technology that was identified as a concern was ease of use of the 
tools embedded in the assessment. The Student Surveys (CBT version) asked the students to 
indicate if it was it easy to use the highlighter and if it was easy to make pictures or words 
bigger or smaller. The results presented in Figures 23 — 24 indicate that more than half of those 
taking the Mathematics assessments did not use the highlighter tool, but of those that did use 
it, the vast majority indicated that it was easy to use. About two-thirds of those completing the 
ELA assessment indicated that they used the highlighter tool, and the vast majority of those 
indicated it was easy to use. 
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Performance-based Assessment: 
Was it easy to use the highlighter? 


m ELA CBT 
n = 221,158 


m@ Math CBT 
33.50% n= 183,099 


% Selected 


4.70% 5.60% 


Yes No Did Not Use 


Figure 23. Percentage of students indicating it was easy to use the highlighter (PBA). 
Percentages are based on the number of valid responses. ELA CBT missing = 1,081; Math CBT missing = 958. 
ELA = English Language Arts; CBT = Computer/tablet-based Test. 


End of Year Assessment: 
Was it easy to use the highlighter? 
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Figure 24. Percentage of students indicating it was easy to use the highlighter (EOY). 
Percentages are based on the number of valid responses. ELA CBT missing = 593; Math CBT missing = 871. 
ELA = English Language Arts; CBT = Computer/tablet-based Test. 
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The results presented in Figures 25 — 26 indicate that only about one-third of students used the 
tool to make pictures or words bigger or smaller, regardless of whether it was a Mathematics or 
ELA assessment. Of those who did use the tool, most indicated it was easy to use. 


Performance-based Assessment: 
Was it easy to make pictures or words bigger or smaller? 


100% 
90% 
80% 
70% 
60% 


m ELA CBT 


50% n= 221,158 
m@ Math CBT 
n = 183,099 


% Selected 


40% 
30% 
20% 
10% 

0% 


Yes No Did Not Change Size 


Figure 25. Percentage of students indicating it was easy to make pictures or words bigger or 


smaller (PBA). 
Percentages are based on the number of valid responses. ELA CBT missing = 1,281; Math CBT missing = 1,093. 
ELA = English Language Arts; CBT = Computer/tablet-based Test. 
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End of Year Assessment: 
Was it easy to make pictures or words bigger or smaller? 


67.20% 65.70% 


mo) 
2 m ELA CBT 
< 50% n = 160,820 
n 

mj Math CBT 
= 20% n= 157,872 


25.50% 


23.70% 


10.60% 


Yes No Did Not Change Size 


Figure 26. Percentage of students indicating it was easy to make pictures or words bigger or 


smaller (EOY). 
Percentages are based on the number of valid responses. ELA CBT missing = 685; Math CBT missing = 1,021. 
ELA = English Language Arts; CBT = Computer/tablet-based Test. 


Finally, the CBT version of the math Student Survey asked students to indicate if the calculator 
tool was easy to use. The results in Figure 27 and Figure 28 indicate that about half of the 
students who responded to the survey indicated that they used the calculator tool. Only about 
10% indicated that it was not easy to use the calculator tool. The results in Figure 29 and 
Figure 30 indicate that more students had problems with entering math symbols and numbers 
than they had with the calculator tool. Across both PBA and EOY, approximately one fourth of 
students reported that it was not easy to enter symbols and numbers into the equation editor. 
Comments from the Student Survey that illustrate difficulty with the equation editor are: 


e When! was typing my equation it did not let me fix my mistake so i had to move on to 
another question and went back and could not change my answer 

e What didn't make sense on the test was putting the fractions in, and soliving the 
equations 

e One thing that made me mad was that when | typed an equation into the computer it 
auto-corrected it for me. The auto-correct changed my answer and it was not what | was 
going to write. 

e While showing my work in the text box | was not able to create a fraction bar. 
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Yes 


Performance-based Assessment: 
Was it easy to use the calculator? 


No 


Math CBT 
n= 183,099 


Quality of Test Administration 


Did Not Use 


Figure 27. Percentage of students indicating that the calculator tool was easy to use (PBA). 
Percentages are based on the number of valid responses. Missing = 1,210. CBT = Computer/tablet-based Test. 
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Figure 28. Percentage of students indicating that the calculator tool was easy to use (EOY). 
Percentages are based on the number of valid responses. Missing = 1,280. CBT = Computer/tablet-based Test. 
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Performance-based Assessment: 
Was it easy to enter math symbols and numbers for your 
answers? 


100% 
80% 67.30% 
60% 
40% 
20% 

0% 


% Selected 


Yes No Did Not Enter Math 
Symbols or Numbers 
Math CBT 
n = 183,099 


Figure 29. Percentage of students indicating whether it was easy to enter math symbols and 


numbers (PBA). 
Percentages are based on the number of valid responses. Missing = 1,212. CBT = Computer/tablet-based Test. 


End of Year Assessment: 
Was it easy to enter math symbols and numbers for your 
answers? 
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Figure 30. Percentage of students indicating whether it was easy to enter math symbols and 


numbers (EOY). 
Percentages are based on the number of valid responses. Missing = 1,169. CBT = Computer/tablet-based Test. 


In summary, the findings indicate the presence of technology-related problems that resulted in 
some interruptions during the testing session. Many TAs reported that they encountered 
technology-related problems during test administration with the most frequently encountered 
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problems being logging onto the assessment and devices locking up (i.e., “stopped working”). 
Converging evidence comes from the observation study; technology-related problems were 
observed in 80% of the schools visited, and problems with logging on and problems with 
devices locking up were also the most frequently observed problems. Two-thirds of the 
observed schools encountered problems that took less than 10 minutes to resolve, while only 
one-third of the observed schools encountered problems that took 10 minutes or more to 
resolve. Overall, approximately half of the students reported that they encountered 
technology-related problems during their test. For students that did encounter problems with 
technology, the most frequently reported problem was with dragging or moving objects on the 
Mathematics PBA. As for the tools (e.g., highlighter, magnifier), of those students who used the 
tools, most reported they were easy to use. Entering math symbols and numbers into the 
equation editor, however, was reported as being difficult for some students. Efforts to address 
technology-related problems should be implemented prior to operational testing to help 
ensure greater adherence to Standard 6.4. Reports such as this are one way to document and 
report disruptions to test administration (Standard 6.3). Actions that PARCC is taking to address 
these concerns are provided in the Executive Summary and in the Summary and 
Recommendations sections under Claim 3. 


Findings for Research Question 7b: Are there any interruptions during the test administration 
related to students asking questions? What do students ask? 


The TA Surveys (CBT and PBT versions) asked TAs to describe any questions students asked 
about the test-taking instructions they read to them. This survey item directly addresses 
Research Question 7b while also addressing the claim that “Students respond to items as 
intended.” Confusion about the instructions and interruptions caused by students asking TAs 
clarification questions about those instructions are both factors that could threaten the veracity 
of this claim. 


The most common themes identified in the TAs’ responses to the survey item asking them to 
describe the kinds of questions students asked about instructions were: 


e What to do when there was an error with the computer? 

e What were the students allowed to do when they finished the test? 

e How to resume the test when they were logged out? 

e What the instructions for specific questions were saying? 

e Howto use or find specific tools (e.g., highlighter, magnifier)? 

e Why certain tools were not available? 

e Whether or not students could continue onto the next session when finished; similarly, 
what to do when they had finished the current session (i.e., submit answers; log out)? 
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Furthermore, although not specific to questions about the test-taking instructions read by the 


TAs, many comments made by the TAs indicated that the instructions were too long or 


repetitive, causing some students to stop paying attention; this may have contributed to some 


of the questions students had regarding instructions. Of the 2,253 TAs who provided a response 


to the survey item asking them to describe the kinds of questions students asked about the 


test-taking instructions (CBT version), nearly half provided comments that were categorized 


into one of the above themes. Some examples of the types of comments provided by TAs were: 


“The instructions were very lengthy - students tended to tune out.” 

“The students did not understand stopping after the 50 minute session was up and 
giving directions for the students finished to end the test. This was confusing for them.” 
“Students were not familiar with some of the tools. For example the icon used to create 
a fraction.” 

“Students had many questions about the Error messages received during the testing.” 


Themes that were specific to the paper-based administration were: 


Switching between the answer document and test document; students were unsure if 
they could write in one or both documents, where to show their work, and how to fill in 
responses using the provided response grid. 

Students in the math sections asked questions about using mixed numbers versus 
decimals, as well as when they could use a calculator. 

Students in the ELA sections asked questions about using a highlighter and how many of 
the passages they were supposed to reference in a constructed response answer. 
Students were confused about directions being read to them, as the page numbers and 
instructions did not match at times. 


Of the 1,076 TAs who provided a response to the survey item asking them to describe the kinds 


of questions students asked about test-taking instructions on the paper-based version of the 


assessment, more than half (approximately 60%) provided a comment that was categorized 


into one of the above themes. Some examples of the types of comments specific to TAs 


administering paper-based versions of the assessments were: 


... there was confusion, both while reading instructions and during the test, about the 
directions for the distribution of calculators during session 1. Also, the instructions did 
not have the students open their answer book, and there was confusion on both days 
when | said to begin because students had difficulty figuring out the answer book and 
did not already have it open to the first page on which they were to answer questions. 
What tools can | use and when? Where/how do | fill in the answer? Numbers? Bubbles? 
Both? Why do! leave my answer as an improper fraction? Do the bullets mean different 
parts of the question? What does justify mean? 
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| tested students with disabilities and those who are English Language Learners, with IEP 
accommodations. Questions students asked were regarding test vocabulary they were 
not familiar with since this was the first time they took a test in this format: reference 
sheet; answer "document"; "stray marks"; "multiple" parts; fraction "bar" 

Can | use graphic organizers in the answer document? What Part A do they mean? 
(when referring to a 2 part questions) How do | mark 2 answers as correct in the answer 
document? Can | refer to all 3 of the articles | read in the constructed response? Why 
does it say only 2? 

When describing how to fill in the answer grid, students struggled to visualize what that 
would look like. A sample problem requiring this type of response would have been very 
helpful. 

Students were very confused on how to fill out the grids for the constructed response 
questions even after we looked at the examples. 

Students were very confused about how to write their answers in the grid. Some right 
justified, some left justified, ignoring the fact that the left-most space was only for a 
negative sign. Other students were confused about where to write the constructed 
responses. 

Students did not understand the (/) to represent the fraction bar. It was too confusing to 
tell them they could fil in the grid from left to right OR right to left. The sentences 
regarding using the grid for mixed numbers and improper fractions was not 
comprehensible. 


Finally, the CBT version of the TA Survey included an additional open-ended question that 


asked the TAs to, “.. . describe any questions students asked about technology-related 


problems.” 


The major themes identified in responses to this open-ended item on the CBT survey were: 


Students had issues logging on, getting logged off before they were finished, and 
questions about how to resume the test; students also had questions about what to do 
when there were internet connectivity issues. 

Students had questions about computer errors, such as the computer freezing, as well 
as questions about java errors. 

Students had questions about how to save answers or retrieve answers deleted by the 
test; for the most part students had to retype these answers. 

Students had questions about using/finding specific tools, or why tools were unavailable 
or malfunctioned. 

Some students who requested accommodations for the test (e.g., text read aloud) could 
not access these tools and the administrator was unable to perform this function for all 
students. 
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e Students had difficulty entering information into text boxes, and some students 
experienced a “lag” in typing that resulted in sentences or entire paragraphs becoming 
mixed up. 


Of the 2,123 TAs who provided a response to the survey question asking them to describe the 
kinds of questions students asked about technology-related problems, nearly half provided a 

comment that was categorized into one of the above themes. Some examples of the types of 
comments TAs provided were: 


e Highlighter had the most questions of any tool. Pre-highlighted sections were very 
confusing to students as well. My booklet of teacher instructions did not cover this, and 
student test did not include directions on what the pre-highlighted sections were until 
they got to that question. Some students somehow accidentally highlighted all of their 
page - questions, text, etc. We were not able to find a way to get rid of the auto - all 
highlighting of a page. | saw two students who had this happen over all of the testing. 
How to indent (without using tab) was another question. 

e They were VERY patient and would raise a hand when there was an issue they knew 
needed attention. They quickly learned how to work through the Java, pop-up blocker 
error messages, loss of connectivity, Early Warnings, logging on and not being at page 
test was supposed to be at and what to do when the reached the end of a section. With 
every error they would ask for assistance. 

e Students had difficulty writing in the text boxes. It would often delete the entire 
paragraph that was written or add italized writing to the text box. Students also were 
having difficulty deleting items within their text box. The backspace key would not work 
on their keyboard or on the screen. 

e Some computers needed to be restarted multiple times in order to allow students to log 
in properly. In some cases students were initially able to open other windows even when 
logged in. This took awhile to resolve so that the students could take the test. 


Finally, findings from the debriefing interviews indicate that 37 schools (61% of schools visited) 
reported that students asked questions during test administration. Those TAs indicated that the 
most common student questions were about: (a) connectivity issues or error messages, (b) how 
to use tools, (c) how to enter answers (particularly fractions, mixed numbers, and exponents), 
and (d) how to answer a question (i.e., navigation and the level of detail expected). 


In summary, findings from the TA Surveys and the observation study indicate that students 
commonly asked TAs questions during test administration. From the observation study, we 
know that students were observed asking TAs questions during testing at over 60% of the 
schools visited. This suggests that some distraction might have been caused by students asking 
questions (Standards 6.3 and 6.4). Findings from the TA Surveys indicate student questions 
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commonly pertained to technology-related issues (CBT only); for example, students asked 
questions about how to handle error messages or how to use the tools. The TAs reported that 
students also commonly asked questions about navigating the test, and they also commonly 
asked for additional clarification on the instructions they read for questions on the test. These 
latter two types of questions suggest that the clarity of instructions could be enhanced 
(Standards 4.16 and 6.5). Additional opportunities for practice on sample items might be one 
way to improve understanding (Standards 4.16 and 6.5). Actions that PARCC is taking to address 
these concerns are provided in the Executive Summary and in the Summary and 
Recommendations sections, most notably under Claims 3 and 4, although improved training of 
TAs (actions pertinent to Claim 1) and improved training resources for TAs (actions pertinent to 
Claim 2) should also help to address the concerns related to Research Question 7b. 


Findings for Research Question 8: If any disruptions, interruptions, or other problems 
occurred, did the TAs deal with the issue appropriately and effectively? 


Research Question 7 focused on interruptions during testing (which is relevant to the claim that 
students respond to items as intended). Research Question 8 focuses on how 
effectively/appropriately TAs handled those interruptions. Thus, Research Question 8 is most 
relevant to the claim that, “TAs are prepared to administer assessments as intended.” For this 
claim to be true, TAs must have the knowledge and skills to handle interruptions, whether 
those knowledge and skills come from background experience and/or from training (Research 
Question 8a). 


As discussed in the results for several of the previous research questions, problems related to 
technology were the most common source of interruptions. Findings from the observation 
study indicate that when technological difficulties occurred the TAs made appropriate attempts 
to resolve the issue by referring to their administration manuals, working with Technology 
Coordinators, and/or contacting the Help Desk. In no instance did the observers note that the 
TAs dealt with the technology problems inappropriately. There were reports, however, of some 
instances in which the technology problems were not resolved. The prevalence and the extent 
of the technology problems suggests that the scope of the problems exceeded the expectations 
for what a TA is expected to handle with regard to resolving basic technology-related problems 
within a testing session. The following comment from a TA on the TA Survey illustrates this: 


e ...Day 1 - Students got some Java Error messages when trying to sign in. Our tech 
person had to call the district tech person and it took over an hour to correct the 
problem before kids could sign in. Then when it was time for 2nd test, we had lots of 
issues regarding the headphones. We discovered if a student was signed in, and the 
headphones weren't already set to a good sound level, when a student tried to adjust 
the headphones, the program immediately quit and logged the student out. Then I'd 
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have to manually resume those students. Students found they needed to adjust the 
headphones to the right level before logging into your program... Day 2 - Session 3 - 
The BIG CRASH. We never got logged on. We tried Internet Explorer. We tried Chrome. 
Nothing worked. | could not even log on to the pearsonaccess website that morning. It 
would just hang. We restarted all computers and nothing changed. The kids that did get 
logged on, had the message that | had to resume them. But | couldn't, because we 
started getting the message that pearsonaccess was not even a website! So | could not 
resume anyone or do anything to help anyone. .. . For the couple of kids who got signed 
in, | gave them the seal code, but then it hung. We spent over an hour trying to log on 
and get the test going, but never had any success. We were instructed by our admin to 
just stop the frustration. So we aborted the field test. 


Aside from handling interruptions caused by technology-related problems, there were two 
sessions observed in the observation study where the observers noted that interruptions were 
caused by student misconduct. In one instance the observer indicated that the TA handled this 
interruption appropriately, and in the second instance the observer rated the TA as having 
“oartially” handled the interruption. The observer notes indicated that there were “some” 
attempts to control the misbehaving student who was talking disrespectfully and being physical 
with other students. Aside from those two instances, there were no observations of TAs 
handling interruptions caused by students’ misconduct. 


The findings presented below under Research Question 8a shed light on whether TAs’ 
preparedness for handling technology-related problems was informed by background 
experience and/or training. 


Findings for Research Question 8a: Were the TAs’ actions informed by background 
experience, by training, or a combination of the two? 


The TA Survey (CBT version) asked the TAs to indicate whether they had ever administered a 
computer-based test to students (either at the state or district level). The results indicate that 
less than half of the 4,635 TAs (46%) who responded to the survey had ever administered a 
computer-based test to students at the district or state level. Consequently, the potential for 
background experience to inform TAs’ handling of technology-related problems was relevant 
for less than half of the TAs. 


The TA Survey (CBT version) also asked the TAs to indicate if they completed the PARCC online 
training modules on topics pertinent to computer-based testing. The results provided in 

Figure 31 indicate that just over half (58%) completed the online training module on 
“Computer-based Testing Test Administration.” Less than one-third (30%) completed the online 
module on, “Accessibility Features and Accommodations with Computer-based Testing.” Only 
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about 10% or less completed any of the other online modules on technology-related topics. 
Moreover, of those who indicated they completed the online training, only about half agreed 
that the training prepared them to resolve basic problems related to technology (e.g., logging 
students in, exiting the test, etc.). 


In summary, findings from the studies indicate that disruptions were most typically the result of 
technology-related problems. Findings from the observation study indicate that the TAs made 
appropriate attempts to resolve the problems by referring to their administration manuals, 
working with Technology Coordinators, and/or contacting the Help Desk. Nonetheless, in 
several instances technology-related problems were not resolved. Less than half of the TAs had 
ever administered a CBT at the district or state level, and only about half indicated that the 
training they received on Computer-based Testing Test Administration helped them to resolve 
basic technology-related problems; these findings suggest that for many TAs their actions to 
resolve technology-related problems were not informed by training or by background 
experience (Standard 6.1). Improved training and greater participation in training might be 
ways to help handle technology-related problems more effectively. Actions that PARCC is taking 
to address concerns related to TA training are provided in the Executive Summary and in the 
Summary and Recommendations sections under Claim 1. Documentation and reporting of any 
such disruptions (Standard 6.3) should also be part of TA training. 
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Which of the online training modules did you complete? 
(select all that apply) 
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Testing based Testing 


Figure 31. Percentage of all TAs responding to the survey who indicated they completed PARCC online training modules. 
CBT = Computer/tablet-based administration. 


Updated February 17, 2015 Page 76 


PARCC Quality of Test Administration 


Findings for Research Question 8b: Would a different approach to handling the problems that 
occurred during test administration been more effective? 


From the observation study, observer notes or TA comments from 10 schools indicated concern 
that not all teachers have the computer savvy to test the students without assistance. This 
concern was based on the frequency of technology problems and the frequency of student 
questions. In 52 of the 61 schools, observers noted there was at least one additional person 
who supported the TA during testing, and across all of the observed sessions there was an 
average of 2.3 staff members administering or helping to administer the field test. In fact, the 
need to have technical staff on hand was expressed by TAs at 11 schools, who noted that extra 
staff was necessary to handle computer preparation and initial sign in support. Schools with the 
most positive experiences indicated that their district technology staff were very involved and 
even created some shortcuts that helped avoid some of the issues regarding Java® and pop-up 
blockers. These findings suggest that extra staff—perhaps extra TAs for handling student 
questions (i.e., non-technology-related questions) and extra TCs for handling technological 
difficulties—may be one approach to more effectively handle problems that arise during test 
administration. Furthermore, the actions that PARCC is taking to address technology-related 
problems from field test, which are provided under Claim 3 in the Executive Summary and in 
the Summary and Recommendations sections, will also help to address problems that arose 
with test administration during field test. 


Findings for Research Question 9: Was security of test materials maintained at all times? 


There is considerable overlap between Research Question 9 and Research Question 6, “Are 
there any apparent attempts to record or copy test materials including the test questions by 
students or others?” Findings that are pertinent to test security were discussed within the 
results section for Research Question 6. Consequently, relevant findings for Research 
Question 9 are reported in the results section for Research Question 6. 


Findings for Research Question 10: Did the test administration create minimal disruption to 
the school and staff? 


For the testing system to be effective, it must not be overly burdensome on the schools’ 
resources. If test administration becomes overly burdensome, then it might not be feasible for 
schools to have sufficient resources and supports to administer the assessments (claim 2). 
While none of the instruments used in the studies explicitly addressed whether test 
administration caused minimal disruptions, from the information gathered across the studies 
we Can gain a sense of the extent to which test administration caused disruptions to the school 
and staff. First, we know from the results discussed for several of the prior research questions 
that technology-related problems, particularly issues with internet connectivity, were present. 
Second, we know from the results discussed for Research Question 8b that staff interviewed in 
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the observation study indicated that extra staff was needed to handle computer preparation 
issues and student log-on issues. Another finding that emerged from the interviews during the 
observation study was that schools with very involved district technology staff were the schools 
that tended to report the most positive test administration experience. These findings suggest 
that disruptions to the school and staff were not minimal, and that considerable resources (e.g., 
strong involvement of district technology staff) were an important factor for successful test 
administration. Also, the number of staff administering or helping administer the field test 
averaged 2.3 per session, which would be a significant tap on resources when testing all 
students. 


Additionally, and as initially described in the results for Research Question 1, during the course 
of the observation study, many comments received during the debriefing interview indicated 
that the training materials for test administration were overly burdensome. For example, 
comments from 19 schools indicated that the administration manuals were cumbersome to use 
and contained information that was not relevant for their role in testing. Typical comments 
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were, “too many manuals, too long,” “would like a summary page or checklist of major steps | 
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have to do,” “...would also like computer screen captures since | am a visual person,” “couldn’t 
easily find the information regarding the number of questions on each test session...and 
timing,” and “better to have chapter summaries.” This was also a common theme mentioned 
by TAs in their response to the open-ended question on the TA Survey about improvements 
PARCC should make to its online training. Example comments provided by TAs on the TA Survey 


were: 


e The webinar is way too long and repetitive. | recommend having the Test Managers 
provide instruction on the information necessary for test administrators. A LOT of time 
was spent after school and before school on preparing us for instruction. It seems like 
the training can be condensed into one session as long as the test administrators read 
the manual. 

e There were many technological problems during the test. There should be a condensed 
version of resolutions to problems that may occur. Each school should have a mock run- 
through of the test so that each administrator knows what to do when a problem occurs. 
Too many problems, too much time! Paper and pencil would have been more time 
efficient. 

e There is so much of a volume of materials to cover (online training/manuals/links) that it 
is difficult to keep up with updates and implementation. One recommendation would be 
to condense some of the information. 

e Condense the information and have screen shots with instructions in the manual. 
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e The set up for Apple computers needs to be simplified. The Java conflicts were quite an 
ordeal to resolve in the weeks before students sat at computers. The number of "man- 
hours" was excessive. 

e Pearson, our school district and the state all told us different things about setting up the 
test. One told us we could only use Mozilla, one only said Explorer, and one only said 
Chrome. Chrome worked but there were several steps to actually get it to work. The test 
coordinator (myself) had to individually go to each student computer to fix it before 
students logged in. This was cumbersome because | had to disable popups, refresh 
pages, and have students type in their information twice. Several times the computers 
logged out of the test and | had to have students restart. | got "test must be resumed by 
proctor” close to 100 times because of errors with the test. The test really needs to be re- 
worked and our state and district leaders need to be better trained on how to relay 
information to test coordinators. There was a lot of mis-communication setting up the 
test. We were not told that the district was doing proctor caching and had to actually 
have a district technology person come out to help us fix the test. We wasted a lot of 
days and student's class time because of this. In my opinion, there needs to be a 
separate Technology Training for Test Coordinators on how to get ready for the test. 
When | called Pearson (who | called on 5 separate occasions), | was told at least twice to 
read the manual. This is not an answer when you have four testing groups of thirty plus 
students waiting to test. 


Collectively, these findings suggest that test administration created disruptions that were more 
than “minimal.” Streamlining the training and training materials, and ensuring that schools are 
supported by a strong district technology staff should help to alleviate some of the burden (It 
should be noted that the burden associated with the technology-related issues is only pertinent 
to schools where tests were administered on computers/tablets). Actions that PARCC is taking 
to address these concerns are provided in the Executive Summary and in the Summary and 
Recommendations sections under Claim 2. 


Findings for Research Question 11a: Did the observers’ responses to the Test Administration 
Observation Checklist adequately and accurately capture relevant aspects of the assessment 
session? 


Overall, the observation checklist worked as intended. The intent of the checklist was to alert 
observers to each task that TAs should complete during test administration. One edit 
recommended for operational use is to clarify the task, “Handle testing irregularities.” More 
guidance needs to be provided to observers on what constitutes “irregularities.” From the 
observer notes, it is clear that some observers included problems with technology as “an 
irregularity,” and others did not. This is evident by the high number of “Not Applicable” ratings 
assigned to this task. In the future, additional training should be provided on this task so that 
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observers are calibrated in their ratings. Another recommendation for operational use is to 
delete the last task on the checklist, “Completes the Test Administrator Survey.” This task was 
completed sometime after the end of test administration, and therefore was not observable 
during the typical test administration observation. 


Two additional recommended edits are to add additional summary fields for observers to enter 
their overall reflections on: (a) the TAs’ knowledge of tasks, and (b) the degree of student 
engagement. A final suggestion for the checklist would be to break some tasks into more 
detailed steps. For example, the task for monitoring student progress using the administration 
computer could target where the computer is located, how often the TA uses it, and if it is used 
only to reset students. Another possible task, dismiss students in accordance with school 
procedures, could target if students were dismissed at one time, or only those not needing 
additional time, or as they completed the session. 


Finally, since it was not known exactly what to expect with the first administration of the PARCC 
field test, observers were encouraged to create detailed descriptions of the sessions they 
observed, including notes of unanticipated issues or events. For operational testing, a more 
succinct set of notes should be encouraged along with a labeling scheme for categorization 
purposes. Also, for operational testing, site observations at schools administering PBTs, which 
were not included in the field test Site Visits Study, will be included. 


Findings for Research Question 11b: Did the test administrators’ responses to the TA Surveys 
adequately and accurately capture relevant aspects of the assessment session? 


Overall, the TA Surveys functioned as intended. The TA Surveys (CBT and PBT) provided useful 
information on: the TAs’ training experience, the clarity of instructions, the pervasiveness of 
different types of technology-related problems (CBT only), the kinds of questions asked by 
students, the use of sample items/tutorials, among other topics. Recommendations for 
revisions to the content of the surveys are to: (a) include an item(s) on the extent to which test 
administration (including preparation for test administration) caused disruptions to the school 
and staff, (b) include an item(s) on the usefulness of the accessibility features and 
accommodations training, and (c) inquire as to whether accessibility and/or accommodation 
features were available for those students for whom they were requested. A final 
recommendation is to streamline the response scales on the surveys (e.g., all items rated on an 
agreement scale) in order to reduce the cognitive load required of persons responding to the 
survey and to simplify data analysis and reporting. 
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Findings for Research Question 11c: Did the students’ responses to the Student Surveys 
adequately and accurately capture relevant aspects of the assessment session? 


Overall, the Student Surveys captured relevant responses regarding important aspects of the 
assessment session. Responses captured in the surveys provided useful information on: 
students’ understanding of instructions, technological issues students faced while completing 
the assessment, usability of tools, whether students had sufficient time to complete the 
assessment, as well as other areas of interest. One edit that is recommended to the content of 
the survey is to include an item(s) assessing students’ request for and availability of accessibility 
features and accommodations. Another recommendation, specific to the paper-based 
administration, would be to include items assessing students’ familiarity with the response grid 
used in PARCC (e.g., had students used this type of answer sheet before? were instructions on 
how to fill-in responses clear?), and the ease with which they could switch from the answer 
booklet to the test booklet (e.g., was it clear to students which document should be used to 
show work?). 


Summary and Recommendations 


The overall summary and recommendations, based on the findings for the research questions, 
are discussed in this section by each claim in the TOA under the Administration phase. Actions 
taken by PARCC to address concerns to test administration are also provided (PARCC, 2014). 


Claim 1: TAs are Prepared to Administer the Assessments as Intended 


In order for this claim to be true important assumptions that must be verified are: (a) TAs are 
trained to administer the assessments and (b) training is effective. Findings related to several of 
the research questions provide evidence for these assumptions and for the overall claim of 
preparedness. 


The research questions that provide evidence related to this claim are: 


e Research Question 2: To what degree do the TAs follow the protocols and instructions? 
(addresses Standards 6.1 and 6.2) 

e Research Question 6: Are there any apparent attempts to record or copy test materials 
including the test questions by students or others? (addresses Standards 6.6 and 6.7) 

e Research Question 8: If any disruptions, interruptions, or other problems occurred, did 
the test administrators deal with the issue appropriately and effectively? (addresses 
Standards 6.1 and 6.3) 

o Research Question 8a: Were the TAs’ actions informed by training, by background 
experience, or a combination of the two? (addresses Standard 6.1) 
o Research Question 8b: Would a different approach have been more effective? 
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e Research Question 9: Was security of test materials maintained at all times? (addresses 
Standards 6.6 and 6.7) 


The investigations of the research questions and their underlying assumptions indicate that 
nearly three-fourths of TAs completed at least one of the PARCC online training modules. Of 
those, about three-fourths completed the Computer-Based Test Administration module—that 
is, 58% of all TAs who responded to the TA Survey for the CBT administration. The next most 
frequently completed online module was the Accessibility Features and Accommodations with 
Computer-based Testing, with 41% completing this module. Very few TAs reported that they 
completed any of the other online modules (i.e., Setting up an Infrastructure Trial/Dress 
Rehearsal, Technology Setup, Student Data Upload, and Emerging Technologies and Security 
with CBT), although it should be noted that the training on Emerging Technologies and Security 
with CBT was not targeted for TAs. 


Training was also provided by schools and districts, and in some rare instances by the state. 
Training provided by the school was the most frequent occurrence with nearly three-fourths of 
TAs administering PBTs indicating they received training from their school and over half of TAs 
administering CBTs indicating they received training from their school. Approximately one-third 
of TAs administering PBTs received training from their district, and nearly one half of TAs 
administering CBTs received training from their district. Overall, these participation rates 
indicate that the majority of TAs participated in some form of training; however, these findings 
suggest that a substantive number of TAs might not have received training, or may have 
received limited training (e.g., completed only one online module when more than one module 
was relevant for the TA). 


Findings also reveal that less than half of the TAs had ever administered a computer-based test 
at the district or state level. Therefore, for the majority of TAs this suggests that their 
knowledge of computer-based test administration likely came from one or more of the 
trainings for PARCC. Although TAs were not asked to indicate whether they had ever taken a 
computer-based test in a proctored setting, it is possible that TAs might have assumed that 
they already knew enough about administering computer-based tests from prior experience 
taking computer-based tests in proctored settings. 


In terms of the effectiveness of the TA training, findings from the TA Survey (CBT version) reveal 
that, of those who participated in online training, three-quarters agreed that it effectively 
informed them of the protocol to follow when administering assessments, and more than half 
agreed that the online training prepared them to resolve basic problems related to technology 
(e.g., logging students in, exiting the test, etc.). These findings indicate that many found the 
online training useful, although there is room for improvement, particularly for training on 
resolving basic technology-related issues. Findings across studies suggest that, in addition to 
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better addressing internet connectivity issues, TAs could benefit from targeted training on 
technology topics such as, logging on and off TestNav, disabling pop-up blockers, use of 
headphones, and use of tools. In addition to providing more targeted and enhanced training on 
these topics, the TAs also recommended that training could be improved by providing 
additional practice with a “mock” test for both students and TAs, which would include practice 
accessing and navigating the TestNav system. Another recommendation was to provide more 
in-person training and to more clearly synthesize all the trainings and training material. Finally, 
given that survey results indicate that there were gaps in training, another important 
overarching goal for helping to improve the preparedness of TAs for administering assessments 
is ensuring greater participation in training. 


Overall, findings from the observation study indicate that most TAs followed most of the 
protocols and instructions in the TA Manual during test administration. However, there were a 
few exceptions. First, in nearly a fourth of the observed sessions the TAs did not post the 
“Testing — Do Not Disturb” sign, although there was only one noted instance of an unauthorized 
visitor/teacher entering the room. Second, in over a third of the sessions the observers noted 
that TAs had not arranged computers so that there were partitions and/or sufficient space 
between computers. (This has implications for test security and thus score interpretation.) 
Third, in approximately half of the observed sessions TAs did not keep track of time using a 
timing box as instructed in the TA Manual. Qualitative evidence from the debriefing interviews 
and from comments from the TA Surveys indicates that TAs had difficulty interpreting the 
manual’s instructions for managing time, and, in particular, the guidance on additional time 
allowed. The confusion about managing time might partly explain why nearly a third of the 
observed TAs did not use the timing box. Finally, the most notable deviation from test 
administration instructions was that in nearly half of the observed sessions the TAs did not read 
the script verbatim. Observer notes indicate that the most common modifications to the script 
were: (a) skipping repetitive parts, (b) changing words (e.g., changing “session” to “part”), and 
providing additional guidance beyond what was in the script (e.g., for logging into TestNav, for 
using headphones). The test administration script is important to ensuring consistency across 
PARCC test administrations. Consequently, variations to the script are a potential threat to the 
validity of PARCC score interpretations. Based on these findings, targeted efforts should be 
made prior to operational testing to ensure that TAs take appropriate steps to protect test 
security; for example, by ensuring that there is appropriate space and/or partitions between 
test-takers’ computers/devices. Furthermore, training and instructions on timing tests, 
including instructions on additional time, should be clarified so that TAs are better able to 
manage time during the test to ensure that all students have the appropriate allotted time. 
Finally, additional training and/or edits to improve the clarity of the script read by the TAs 
should be conducted to ensure that TAs are consistent across sites. 
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In terms of disruptions to test-taking and TAs’ handling of those disruptions (Research 
Question 8), problems with technology represented the most notable disruption. Findings from 
the observation study indicate that, overall, the TAs handled the technology problems to the 
best of their ability. For example, they consulted their TA Manual for assistance, they worked 
with their Technology Coordinators, and they contacted the Help Desk for assistance. 
Nonetheless, in some instances the technological issues were not minor, and in some cases the 
issues were not resolved despite the TAs’ best efforts. In terms of what could have been done 
differently, TAs expressed the need for additional support from technical staff to help prepare 
for and deal with the technology issues. Schools that reported the most positive experiences 
were those that indicated they had a very strong district technology staff to assist them. 
Consequently, another recommendation to better prepare for test administration is to have 
more support from technology staff. 


In summary, the evidence from these research studies for Claim 1 indicates that the majority of 
TAs participated in some form of training (e.g., on-line module(s), school-led training, district- 
led training). The majority also agreed that training was effective for informing them of the 
protocol to follow when administering the field test, and for preparing them to resolve basic 
problems related to technology. Furthermore, in the observation study, most TAs were 
observed following most of the protocols in the TA manual. This evidence represents a solid 
base on which to build for operational testing. There was also evidence, however, that indicates 
potential threats to the validity of the claim that “TAs are prepared to administer the 
assessments as intended” and the underlying assumptions that “TAs are trained to administer 
the assessments” and that “training is effective.” There were substantive numbers of TAs who 
did not participate in the various trainings. For those who did participate in training, the 
findings indicate that there is room for improvement in the effectiveness of training, 
particularly with regard to training on resolving basic problems with technology. When 
technology problems arose, most TAs appropriately consulted their TA Manual, Technology 
Coordinators, and/or the Help Desk for assistance, but the technology problems occurred to an 
extent that exceeded the capacity of those resources to effectively resolve all problems ina 
timely manner. Furthermore, nearly half of the TAs observed in the observation study did not 
follow the script completely in the TA manual, which poses concerns for consistent test 
administration across sites. These threats to the claim that “TAs are prepared to administered 
the assessments as intended,” undermine the validity of subsequent claims in the TOA related 
to scoring, reporting, and use of PARCC scores. Consequently, steps should be taken to address 
these threats by considering the recommendations for improving training and preparedness 
discussed above. 
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In light of the information gathered during field testing, PARCC is already taking several actions 
to address concerns and potential threats to the claim that “TAs are prepared to administer the 
assessments as intended.” Actions that are being implemented by PARCC to improve the 
preparedness of TAs include: 


e Providing TA training modules earlier in the school year to facilitate advance 
preparation. 

e Splitting modules into task based roles so that Test Administrators and Test 
Coordinators can clearly and easily identify which tasks they are responsible for; 
manuals include separate symbols for TA tasks and TC tasks for clarity. ® 

e Splitting up training modules into chapters by topic with search fields to allow easier 
navigation and to facilitate locating information. Providing a summary of where to find 
key training information. 

e Enhancing the guidance and documentation around the SystemCheck tool to ensure 
wider use and understanding. 

e Revising tutorials to include the full array of tools, accessibility features, and item- 
computer interactions. 

e Updating the Accessibility Features and Accommodations training module, and updating 
the PARCC Accessibility Features and Accommodations Manual. 

e Expanding practice tests to include paper and to include both components (PBA and EOY 
in both content areas) so that TAs can become more familiar with the tests prior to 
administration. 


These actions help to ameliorate concerns identified during the field test, and will help to 
ensure that TAs are better prepared to administer the PARCC assessments during operational 
testing. 


Claim 2: TAs Have the Resources and Supports to Administer the Assessments 


For this claim to be true important assumptions that must be met are: (a) resources and 
supports must be clear, sufficiently detailed, and easy to follow, and (b) resources and supports 
must not be overly burdensome for TAs to use and apply. 


*8 The PARCC states have made several improvements to available modules based on field test feedback, including: 
Student Registration Import (September 19, 2014); PearsonAccess"™ (October 9, 2014); Paper-based Testing for 
Test Coordinators (October 10, 2014); Paper-based Testing for Test Administrations (October 13, 2014); 
Introductions to Training (October 20, 2014); Accessibility Features and Accommodations (October 21, 2014). 
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The research questions that provide evidence for this claim and these assumptions are: 


e Research Question 1: To what degree do the test administrators (TAs) find the 
instructions clear, sufficiently detailed, and easy to follow? (addresses Standard 4.15) 

e Research Question 10: Did the test administration create minimal disruption to the 
school and staff? 


The investigations of the research questions and their underlying assumptions indicate that 
more than two-thirds of TAs agreed that the policies and procedures in the TA Manual were 
easy for them to understand. Slightly fewer agreed that the instructions (including the scripts 
for administering the assessments) were easy to implement (63% and 55%, respectively for CBT 
and PBT). Even though the majority of TAs indicated that the TA Manuals were easy to 
understand and implement, a substantive number (approximately 30% — 45% across CBT and 
PBT) disagreed that the TA Manuals were easy to understand and implement. Feedback 
provided by the TAs indicates that there were some inconsistencies between what appeared in 
TAs’ scripts and what appeared on students’ screens or booklets (e.g., script indicates that 
information is located in one place, but information appears in a different place on students’ 
screens/booklets, screens have a ‘Continue’ icon, but script instructs students to ‘Stop’ before 
hitting continue). Such disconnects contributed to some confusion in navigating through the 
tests. The TAs also indicated that the instructions for submitting answers and completing tests 
(CBT version) were not easy to understand. Furthermore, TAs indicated that the TA Manual was 
confusing because large parts of the manual were not relevant to the particular grade being 
tested. This required the TAs to skip various sections of the lengthy manual to find the sections 
that were relevant to the grade being tested; TAs reported that having to skip around made it 
easy to get lost in the script. The TAs also specifically mentioned that the instructions for timing 
the test, including the instructions for additional time allowed, were unclear. Another specific 
point of confusion for TAs was the use of the terms “sections” and “sessions,” which were used 
interchangeably and inconsistently. Some TAs also reported confusion about the read aloud 
accessibility feature/accommodation (text-to-speech vs. human reader) described in the 
Accessibility and Accommodations Manual. Finally, some points of confusion that were specific 
to the paper-based administration were: (a) no page numbers in the student instruction books 
(students did not know which page to turn to), and (b) confusion about the instructions for 
when calculator use was allowed and not allowed (was allowed for some part(s) within a given 
section, but not all parts in that section, which made it difficult for TAs to monitor). Some of 
these issues with the TA Manual and the script might help to explain why deviations from the 
script were observed in the observation study. These sources of confusion should be rectified in 
order to help reduce confusion, and thereby help ensure consistent implementation of the 
protocols and instructions across test administrations. 
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Findings from the research studies also indicate that the “volume of materials” (e.g., online 
training, manuals, and various communications and updates), and the time required to 
synthesize and digest all that information was overly burdensome. This might also partly 
explain why there were issues during test administration; if TAs were overloaded and 
overwhelmed by the plethora of materials and information, they may not have been able to 
digest, comprehend, and implement all of the information for a successful test administration. 
A common recommendation was to condense and synthesize the training and the training 
materials into more a digestible amount and format. TAs recommended that there should be 
grade-specific scripts and that there should be a summary or checklist of major steps and key 
information. 


In addition to evidence indicating that the training materials and information were overly 
burdensome, findings also indicate that the amount of time and effort required of school and 
district technology staff to ensure technology requirements were in place was “excessive” 
(many of the TAs who participated in the studies served dual roles of TAs and TCs). Moreover, 
even with “excessive” time and effort spent preparing the devices, technological problems 
during test administration were still present during administration. Nearly 90% of TAs reported 
some technology issue. This evidence indicates that testing created disruptions that were more 
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than “minima 


In summary, the evidence collected from the studies indicates that there are threats to the 
validity of the claim that “TAs Have the Resources and Supports to Administer the 
Assessments.” Those threats stem from the moderate to weak support for the underlying 
assumptions that: (a) resources and supports (defined as training materials in these research 
studies) are clear and easy to implement, and (b) resources and supports are not overly 
burdensome for TAs to use and apply. These threats undermine the validity of subsequent 
claims in the TOA related to scoring, reporting, and use of PARCC scores. Consequently, steps 
should be taken to address these threats by considering the recommendations for improving 
the training materials discussed here. 


Actions by PARCC to Address Concerns Related to Claim 2 


In light of the information gathered from field testing, PARCC is already taking several actions to 
address concerns and potential threats to the claim that “TAs have the resources and supports 
to administer the assessments.” The “resources” and “supports” (beyond the trainings 
discussed under Claim 1) that were the focus of these research studies were the TA Manual 
protocols and script. Actions that are being implemented by PARCC to improve the TA Manual 
are: 
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e Shortening the TA Manual by reducing redundancies so that there is not as much 
information to sort through. 

e Working with grade level experts to ensure administration scripts are grade appropriate, 
as scripts are being split into grade bands within the TA manual to facilitate their use. 

e Updating the TA manual to include simple and direct checklists of tasks to complete 
before, during and after testing. 

e Updating the TA manual to include more explicit directions, including graphics and icons 
to increase user understanding. 

e Changing the terminology for testing sessions to clarify terms for testing. 

e Clarifying instructions for calculator use to make it easier for TAs to monitor calculator 
use; for CBT, changes are being made to the computer-based interface to make it easier 
to see if the student is in a calculator section; for PBT, a calculator icon will be added to 
the pages in the test booklet to make it easier for TAs to monitor. 

e Updating instructions in the TA manual so there are no disconnects in directions for the 
“Submit” function and the “Exit & Save” function. 

e Simplifying the testing time policies described in the TA manual so that it is easier for 
TAs to understand how to manage testing time. 

e Sending the TA manual to PARCC and state representatives for review to ensure that 
procedures and directions are simple and clear. 

e Updating the Accessibility Features and Accommodations Manual. 


The above actions, coupled with actions to streamline and simplify resources, help to 
ameliorate concerns identified during field testing with the PARCC resources and supports. 


Claim 3: Technology Improves and Facilitates the Assessment Experience 


For this claim to be true an important assumption that must be met is that the technology must 
work as intended. If the technology does not work as intended—for example, if there are 
functionality problems and/or delays in its application, then the veracity of this claim is 
threatened. 


The evidence for this claim comes directly from Research Question 7a: 


e Are there any hardware or software related problems during the test administration? 
(addresses Standards 6.3 and 6.4) 


Findings from the research studies indicate that many TAs reported that they encountered one 
or more problems related to technology during the assessment administration. Students 
encountered problems with the technology at a much lower rate. Approximately half of the 
students indicated that they encountered technology-related problems during their test. 
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Reports of technology-related problems might have been lower for students than for TAs 
because students were asked about problems that occurred during their own individual test, 
whereas TAs were asked about problems across all students in their administration session(s). 


The TAs reported that they most frequently encountered problems with logging into the 
assessment. The next most frequently encountered problems reported by TAs were with 
devices that stopped working (“shutting down”), devices working slowly, and losing internet 
connection, respectively. Students, on the other hand, reported higher occurrences of devices 
working slowly than of devices “stopping” (i.e., shutting down) entirely. Some students also 
reported difficulties with dragging or moving objects on the screen and difficulties making 
changes to answers. Difficulty making changes to answers was reported at a higher frequency 
for the Mathematics tests than for the ELA tests. Some students also reported difficulty 
entering math symbols and numbers into the equation editor on the mathematics assessments. 


Converging evidence of technological difficulties came from the observation study in which 
many of the schools visited were identified as having problems with technology. In terms of the 
severity of the technological difficulties, findings indicate that problems categorized as “minor” 
(i.e., requiring less than 10 minutes to resolve) occurred at two-thirds of the observed schools, 
and problems categorized as “severe” (i.e., requiring 10 minutes or more to resolve) occurred 
at one-third of the observed schools. The most frequently observed technological problems 
were with logging students into the assessment and with devices shutting down. 


In summary, findings from the research studies provide converging evidence that technological 
problems were an issue. Consequently, the evidence does not widely support the validity of the 
claim that “technology improves and facilitates the assessment experience.” Because the 
evidence collected indicates a lack of support for this claim, this undermines the validity of 
subsequent claims related to scoring, reporting, and use of scores. One positive finding that did 
emerge from the research studies, however, is that there did appear to be a slight reduction in 
the occurrence of the technological problems from the time in which the initial PBAs were 
administered to the subsequent administration of the EOY assessments (though the PBAs may 
have included more innovative technological enhancements than the EOY assessments). This 
suggests that improvements were made over time, and that problems can be addressed and 
reconciled for future administrations. Feedback provided from the TA Survey and debriefing 
interviews indicates that one way to help deal with technological related issues is to have 
strong support from district technology staff. Enhancements to technology-related training for 
TAs, additional practice opportunities (see discussion of recommendations under Claim 1), and 
fine-tuning the PARCC SystemCheck tool to improve identification of technical difficulties prior 
to testing are additional ways to help address technology-related problems. Another positive 
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finding from the Student Surveys was that, of those students who reported using the 
highlighter tool and the magnifying tool, most indicated that they were easy to use. 


Actions by PARCC to Address Concerns Related to Claim 3 


In light of the information gathered from field testing, PARCC is already taking several actions to 
address concerns and potential threats to the claim that “Technology improves and facilitates 
the assessment experience.” Although local districts are responsible for their own technology 
(e.g., devices, internet connectivity, etc.), the PARCC consortium is working to provide districts 
and schools with additional supports to improve the local technology experience. Actions that 
are being implemented by PARCC to address technology-related issues include: 


e Improving exiting procedures in the test delivery system so that students do not get 
locked out of tests prematurely. 

e Conducting an extensive verification and validation process check of the delivery system 
prior to the operational assessments to confirm that the system can support the 
thousands of simultaneous test-takers without interruptions and without loss of student 
answers. 

e Working with a vendor to conduct a large-scale “dress rehearsal” to identify issues with 
the highlighter tool and other tools (e.g., magnifier and drag-and-drop) and develop 
solutions to further improve functionality before operational testing. 

e Reviewing and improving information shared with school districts and schools about the 
technology requirements and about how to set up devices to reduce blockage of videos 
due to filter settings on testing devices. 

e Providing more time for the infrastructure trial by making it available to schools earlier. 

e Enhancing guidance and documentation around the SystemCheck tool to ensure wider 
use and understanding of the tool. The SystemCheck tool allows users to validate that 
their testing workstations meet the minimum requirements needed to run TestNav8 for 
the assessments and evaluate bandwidth capacity for internet and proctor-caching 
connections. 

e Updating the training modules available to guide users in conducting an infrastructure 
trial and proctor caching to ensure that the modules are more comprehensive and 
provide clearer step-by-step instructions. 

e Providing a new technology preparedness training module to help organizations 
understand, manage, and make decisions in preparing school technology to be used for 
online testing. 


"t) to make it more 


e Launching an updated interface for PearsonAccess (PearsonAccess 
user-friendly. 
e Improving the equation editor in the Mathematics assessments so that it is easier for 


students to use. 
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e Providing practice tests to allow students and teachers to become more familiar with 
the equation editor and other tools. 

e Increasing the knowledge of Call Center staff so that they are better equipped to help 
TAs resolve technology-related problems. 

e Increasing the number of representatives staffed at the Call Center. 

e Creating a “Quick Start Checklist” to have a cohesive step-by-step guide to start up. 

e Adding a live chat feature to the Customer Support Center to allow additional avenue 
for responses. 

e Linking the call-in process to PearsonAccess"™ user profiles to limit time spent 
identifying callers. 

e Bringing in an independent vendor to test the platform and identify performance issues 
needing improvement. 

e Conducting “load testing” and other tests to pinpoint performance issues and solutions. 

e Improving information to the field on minimum technology specifications and how to 
set up devices. 

e Developing clear, consistent, and comprehensive communication to be sent to states 
and districts regarding technology preparedness tools, resources, timelines, and 
support. 


These actions help to ameliorate concerns identified during the field test, and will help to 
ensure that technology improves and facilitates the assessment experience. 


Claim 4: Students Respond to Items as Intended 


Finally, for this claim to be true, several assumptions must be verified. First, for students to 
respond to items as intended, they must understand the directions read by the test 
administrator (i.e., the script). Second, students must also understand the directions they read 
for the questions on the test. If the students do not understand the directions read by the TA 
and/or if they are confused by the directions they read on the test, then they may not respond 
to the items as intended. Third, to respond to items as intended, students need to have some 
engagement with the assessment. If they are not engaged or if they have a low level of 
engagement, then they might not sufficiently attend to items so as to respond to them as 
intended. Several factors might negatively impact student engagement, such as distractions 
caused by: (a) unauthorized visitors, (b) disruptive student behavior, and (c) technological 
problems, to name a few. 


The research questions that provide evidence for this claim and these assumptions are: 


e Research Question 3: Do the students appear to understand the instructions provided to 
them? (addresses Standards 4.16 and 6.5) 
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e Research Question 4: To what degree are students engaged in taking the test? 
(addresses Standards 4.16, 6.3, 6.4, and 6.5) 
e Research Question 5: Is there any disruptive student behavior during the session? 
(addresses Standard 6.4) 
e Research Question 7: Are there any interruptions during the session? (addresses 
Standards 6.3 and 6.4) 
o Research Question 7a: Are there any hardware or software related problems 
during the test administration? 
o Research Question 7b: What type of questions, if any, do students ask during the 
test administration? (also relevant to Standards 4.16 and 6.5) 


Findings from the Student Surveys indicate that the majority of students (78% or more across 
the test types and administration modes) report that they understood the directions (i.e., 
script) read by their TA. This evidence helps to verify the assumption that students understand 
the directions. However, there was also converging evidence from the observation study and 
from the TA Surveys that the script read by the TA was too lengthy, and that, as a result, some 
students quit listening/paying attention to the directions read by the TAs. If students stopped 
paying attention to the TAs’ script—that is, they were not engaged—then, they might not have 
fully understood the directions, which is a threat to the validity of the claim that students 
respond to items as intended. One recommendation to address this threat is to shorten the 
length of the TA scripts. A recommendation that surfaced from the research studies was to 
shorten the scripts by removing repetitive content, particularly when TAs are administering 
back-to-back sessions. 


Next, findings from the Student Surveys indicate that the majority of students had some 
difficulty in understanding the directions they read for the questions on the test. This was 
particularly true for the Mathematics tests with only about 15% of students indicating that it 
was hard to understand the directions for the questions on the tests “almost never.” For ELA 
tests, about a third of the students indicated that it was hard to understand the directions 
“almost never.” These findings reveal that the majority of students had difficulty understanding 
at least some of the directions for questions on the test. Indeed, when students asked their TAs 
questions, findings from the TA Surveys indicate that students often asked questions, such as, 
“What is this question saying?” If students are confused about what the test items are asking 
them to do, then they might not be responding to the items as intended. One recommendation 
that surfaced from the research studies to help ameliorate this concern was to give TAs and 
students more exposure to practice items. 


In terms of student engagement, findings from the research studies reveal that the TAs 
effectively kept distractions such as, outside visitors and disruptive student behaviors to a 
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minimum. However, even though disruptive student behavior did not appear to be much of an 
issue, there were some reports from TAs that students finished the test early and became 
“fidgety” and distracting to the students who were still testing. One recommendation was to 
allow students to read a book as soon as they have completed their test. 


The biggest distraction, and thus impediment to student engagement, appears to have been 
the technology-related problems, which oftentimes caused delays in testing and/or resulted in 
the loss of student data. As a consequence of the technology problems, some students 
reported that they skipped items that were not functioning correctly or that they rushed to 
finish the test because of the delays caused by the technology problems. If students are rushing 
through the test to finish on time and/or skipping items, then student engagement is negatively 
impacted, which means students might not be responding to items as intended. Earlier 
recommendations discussed above to provide TAs with more targeted training on handling 
technology-related problems and providing schools with more support from technology staff 
should also help alleviate concerns with student engagement stemming from technology- 
related issues. 


Overall, findings from the research studies indicate that there are threats to the validity of the 
claim that students respond to items as intended. On a positive note, there were some reports 
from TAs and students that, aside from the technology problems, they liked the interactive 
nature of the computer/tablet-based tests. This suggests that if technology problems are 
resolved, student engagement should increase. 


Actions by PARCC to Address Concerns Related to Claim 4 


In light of the information gathered from field testing, PARCC is already taking several actions to 
address concerns and potential threats to the claim that “Students respond to items as 
intended.” Many of the actions already identified under Claims 1, 2, and 3 will help to ensure 
that students respond to items as intended. Additional actions not already addressed above 
that PARCC is taking to help ensure that students respond to items as intended are: 


e Expanding tutorials to include the full array of tools, accessibility features, and item- 
computer interactions.” 

e Revising tutorials to make them more grade-appropriate. 

e Expanding practice tests to include paper and to include both components (PBA and EOY 
in both content areas) so that students can become more familiar with the tests prior to 
administration. 


*° The planned release dates for the expanded student tutorials for CBT and PBT are: High school tutorials: 
November 2014 (available at parcc.pearson.com/tutorial); Grade 3 — 8 tutorials: January 2015 (available at 
parcc.pearson.com/tutorial). Online practice tests for the mathematics EOY assessment and the ELA PBA 
assessment are currently available. 
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e Updating the test scripts in the manual to contain more grade-appropriate language, 
which should help to improve students’ attention to the scripts and understanding of 
the information conveyed in the scripts. 

e Revising general directions on the tests, especially the mathematics tests, to make them 
clearer. 


These actions combined with the actions described under Claims 1, 2 and 3 will help to address 
threats to the claim that students respond to items as intended. 


Conclusion 


The findings presented in this report do not provide an exhaustive investigation of the validity 
evidence for the PARCC test administration. Consequently, the evidence presented here should 
not be taken as the final word on the PARCC field test administration. Nonetheless, the findings 
presented in this report do represent the foundation of the validity argument for the PARCC 
field test administration. While there were some pockets of positive findings (e.g., some 
positive comments about the interactive nature of the CBT, reports that the tools were easy to 
use, and TAs following the protocol, in most instances, as instructed) there is evidence 
indicating a lack of support for the claims in the administration phase in the TOA. Because lack 
of support for earlier goals may undermine the validity of subsequent goals, the evidence 
reported here represents concerns for the validity of PARCC test score interpretations from 
future operational assessments for their intended uses if these issues persist in future 
administrations. The field test investigations described in this report were useful and helpful 
endeavors for identifying threats to valid score interpretations that arose during field test 
administration. The actions already being implemented by the PARCC consortium will help to 
ensure that those threats are reconciled prior to operational testing. The planned Test 
Administration Study for operational testing (i.e., TA Survey, Student Survey and Site 
Observations) will help to verify whether these corrective actions have been effective. This 
validity argument should be updated as new evidence is collected. 
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Appendix A — Observation Checklist for Test Administration Site Visits 


PARCC Field Test PBA and EOY Computer-Based Assessment Observation Checklist 


Date: 


School: 


TA/TC: 
City: 


1=M 


State: 


Step Task 


Prepare testing room. (every other seat or partitioned, room configured 


Pre-Assessment Activit 


Prepare student computers for testing. (close browsers and programs, 
launch TestNav) 
Prepare Test Administrator computer. 


ies 


Status Codes: 


et 


2= Partially Met 
3=Did Not Met 
4=Not Observed 
5 =Not Applicable 


Status 


Session# & Time: 
(ex: 0900, 1530) 


Grade: 
Observer: 


Comments/Notes specific to line item 


3 
for security/privacy) 
4 |Post "Testing - Do Not Disturb" sign on outside door(s). 
5 Ensure prohibited materials are not visible. (posters, maps, number lines, 


word lists, formulas) 


Allow authorized visitors only, such as PARCC staff, state/local/school 
assessment monitors. (List all visitors) 


Notes: 


a_ |Student Authorization Ticket. 


TA confirm correct Student Authorization Tickets and seal codes for this 
Draw a timing box on the board as provided in script. (see examples on tab 
or last page of observation form) 


Administer Assessment (complete for both regular and accommodation sessions) 
electronics, or reference books. 
. 
advises. 
: 


Scratch paper-2 pages (blank, lined, or graph paper) and pencil. Students 
can request more. Students print name in upper right-hand corner of paper. 


Hand-held calculators (TestNav supplied or approved): 
c |¢ Grades 3-5 - students with calculator accommodation only 
¢ Grades 6 to HS - for calculator portion of test only 


Tracing paper, reflection tools, straight edge, and compass are allowed for 
grade 8students and those taking Geometry and Integrated Math only. 


Headphones: Required for PBA ELA test sessions but not EOY. Additionally, 
students with accommodations may use headphones. 

NOTE: Student can use headphones as noise buffer, in which case the TA 
must ensure they are not plugged in. If TA does not verify, rate as "Partially 
Met" and provide a comment. 


12 |Read script verbatim. (can repeat, but not modify) 


Notes: 
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Keep track of time during the test session (see tables on tab or last page of 
checklist) by updating the timing box and providing a verbal prompt of 
time remaining as noted in the script. 


13 


Ensure students log in and have access to the test. 
Monitor student progress using administration computer. 


Monitor students by circulating around the room. 


Answer student questions following script guidance. (note questions) 


Provide up to 50% additional time if needed. (note how many students and 
how much extra time was allowed) (rate 5-N/A if accommodation session) 
Provide a group 3 minute break if necessary (computer screens must be 
turned off or apply visual block to monitor). Students must not 
communicate during breaks. Individual restroom breaks are permitted. 


Handle testing irregularities. (e.g. student misconduct; cheating; using 
electronic devices; fire drills or safety threat; loss of Internet connectivity) 
Allow students finishing the test early to check their answers, sit quietly, 
or read for pleasure. (MD, NM, and TN: students can also be dismissed) 


Maintain test security if test is interrupted. (no interruption, rate 5-N/A) 


Quality of Test Administration 


Notes: 


Ensure students receiving the same accommodations are grouped 


together. 


Accommodations (Special Education or English Learner) 


Provide the number of students in accommodation session. 


Indicate if the students receiving the accommodation(s) are Spec Ed or EL. 


List the accommodations being provided. 


Observers: use accommodation job aid while observing session, note discrepancies. 


Students log off of the session. 


Post-Assessment Activities 


Collect Student Authorization Ticket and any test materials. (including all 
scratch paper 


29 |Dismiss students in accordance with school procedures. 


Additional Comments /Notes: 
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30 |Return testing materials to the Test Coordinator. 
31 |Completes the test administrator survey. (we may or may not observe) 
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Appendix B — Debriefing Protocol for Test Administration Site Visits 


Test Coordinator and/or Administer Debriefing Protocol 


Observer Note: Be sure to provide a note for those statements below that are about previous 
sessions. 


1) How do you feel the testing went today? 


1a) Follow up: If responds positive or negative, ask in what ways. (If problems or issues 
occurred, ask about them specifically). 


2) If students asked questions during the assessment; can you recall what the students asked? 
(skip if there were no question) 


3) If you viewed the training modules, did you find them helpful? 
4) Were the test admin manuals you were provided beneficial in preparing for testing? 
5) Was the training you received beneficial in preparing for testing? 


5a) Follow up: Describe how training or test administration manuals could be improved or what 
was missing that would have helped. 


6) Were sufficient materials and support provided in preparing the computers for testing? (If TA 
does not know, ask Tech Coordinator if possible) 


6a) Follow up: Describe how could they be improved or what was missing that would have 
helped. 


7) Are there any additional comments you would like to provide regarding the test 
administration, testing materials, training materials, or support? 


8) Accommodation Only: Prior to giving the accommodation (e.g. read aloud, human scribe, 
interpreter, etc.), what kind of training did you receive? 
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Appendix C — Test Administrator Survey for 


Computer/Tablet-based Administration 


PARCC 2014 Field Test 


Thank you for administering the PARCC Field Test. Now, we want to ask you some questions about 
the assessment(s) you just administered. We will use your answers to help improve the test 
administration process. 


* Please indicate the mode of assessment administration: 
© Computer-based (or tablet) administration 


© Paper-and-pencil administration 


Page 1 
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PARCC 2014 Field Test 


(Background) 
1. Please identify the location where you administered the assessment: 


6 
© Arkansas 


4 


a 
= 
‘ 
a 
P 
e 
FS 
* 


»9aAana #4 


* 2, Please enter the name of the LEA/District in which you administered the assessment. 
LEA/District Code: 

District Name: 

* 3, Please enter the name of the school in which you administered the assessment 


School Name: 
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PARCC 2014 Field Test 


4a. Please identify the grade level(s)/course(s) in which you administered the assessment 
(s) (select all that apply). 
Abbreviation Definitions: 
ELAIL PBA = English Language Art/Literacy Performance-Based Assessment 
ELAI/L EOY = English Language Art/Literacy End of Year 
ELAIL PBA ELA/L EOY 
c oS 


i Cc 
Cc r 
Cc a 
G a 
os Cc 
Cc cr 
Cc 
rc Ci 
Cc Cc 
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PARCC 2014 Field Test 


4b. Please identify the grade level(s)/course(s) in which you administered the assessment 
(s) (select all that apply). 
Abbreviation Definitions: 
Math PBA = Math Performance-Based Assessment 
Math EOY = Math End of Year 
Math PBA 


Not Applicable - | did r 
not administer a Math 
Assessment 


Grade 3 
Grade 4 


Grade 6 
Grade 7 
Grade 8 
Algebra 1 
Geometry 
Algebra 2 
integrated Math 1 
integrated Math 2 
integrated Math 3 


(Test Administrator's Training Experience) 


Lh ORL GO Oe OF eo Oe oO 
Ce Oe ORE OPC Oo Oa oO 


1. Have you ever administered a computer-based test to students before (either at the state 
or district level)? 
© Yes 


© Ne 


2. Did you complete any of the online PARCC training modules (as covered in the online 
training found here https://parcc.tms.pearson.com))? 


© Yes 


© No 
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2a. Please indicate which of the online training models you completed (select all that 


Accessibility Features and Accommodations with Computer-Based Testing 
Paper-Based Testing Test Administration 
Computer-Based Testing Test Administration 
Setting Up An Infrastructure Trial/Dress Rehearsal 
Technology Setup 
Student Data Upload Overview 
Emerging Technologies and Security with Computer-Based Testing 
2b. The PARCC online training(s) effectively informed me of the protocol to follow when 
administering the assessment. 
© Not Applicable 
© Strongly Disagree 
c Disagree 
© Agree 
© Strongly Agree 
2c. The PARCC online training(s) prepared me to resolve basic problems related to 
technology (e.g., logging students in, exiting the test, etc.) 
© Not Applicable 
© Strongly Disagree 


c 
c 
c 
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PARCC 2014 Field Test 


2d. Do you have any specific recommendations on how PARCC can improve its online 
training? 
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PARCC 2014 Field Test 


3. Did you receive any other training, aside from the online PARRC training modules? 
(select all that apply)? 


™ Yes, | received training from my state. 
™ Yes, | received training from my district 
[™ Yes, | received training from my school. 
[ No, | did not receive any training from my state, district or school 


(Test Administration Experience) 
Please rate your level of agreement with each of the following statements (1-4). 


1. The policies and procedures within the Test Administrator Manual were easy for me to 
understand. 


© Strongly Disagree © Disagree © Agree © Strongly Agree 
2. The instructions (including scripts for administering the assessment) within the Test 
Administrator Manual were easy for me to implement. 

Cc Strongly Disagree c Disagree Cc Agree Cc Strongly Agree 
3. Students appeared to understand the instructions I read to them during test 
administration. 

c Strongly Dis agree c Disagree Cc Agree © Strongly Agree 
4. The instructions | read to the students covered all of the information necessary to take 
the test. 

© Strongly Disagree © Disagree C Agree © Strongly Agree 
5. Please use the space below to describe any questions students asked about the test- 
taking instructions you read to them. 


«| 
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PARCC 2014 Field Test 


6. Please identify the types of technology-related problems that occurred during test 
administration (select all that apply). If that problem occurred, indicate how it was resolved. 
Indicate whether this 
problem occurred 


Resolved problem on Resolved problem with Problem was not 
my own assistance resolved 


rc Bi eS 


during the test. 


7. Please use the space below to describe any questions students asked about 
technology-related problems. 


(Test Environment) 


1. Were there any problems in the physical environment during the test session that might 
have negatively affected student performance? 


© Yes 


© No 
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PARCC 2014 Field Test 


1a. Please indicate the nature of the problem in the physical environment. (select all that 
apply) 


™ Computers/devices were not arranged in an appropriate way to facilitate student work 
™ Lighting was insufficient 
[™ Other (please use the space below to identify ‘other’ issue) 


Other (please specify) 
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PARCC 2014 Field Test 
(PARCC's Timing Policy) 


(PARCC’s Timing Policy) 


1. Indicate the proportion of students who needed the Additional Time Allowed. 
© None 
© Less than a third of students 
© Approximately half of students 
© More than half of students 
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PARCC 2014 Field Test 
(PARCC’s Timing Policy) 


1a. Please indicate how much additional time was used by the majority of these students. 
© Less than 10 minutes 
© Between 10-20 minutes 
© More than 20 minutes 


Page 11 
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(Accessibility Features and Accommodations) 


1. Are you aware there is an approved PARCC Accessibility Features and 
Accommodations Manual? 


© Yes 


© No 
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PARCC 2014 Field Test 


1a. Did you read the “Accessibility Features and Accommodations Manual” prior to 
administration? 


© Yes 


© No 
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PARCC 2014 Field Test 


2. Are you aware there is approved PARCC guidance on “Computer-Based Accessibility 
Features and Embedded Accommodations"? 


© Yes 


© No 
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PARCC 2014 Field Test 


2a. Did you read the “Computer-Based Accessibility Features and Embedded 
Accommodations Available for PARCC Field Test”? 


© Yes 


© No 
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PARCC 2014 Field Test 


3. Indicate whether you administered any of the following accessibility features or 
accommodations to your students during the Field Test administration? (select all that 


| did not administer any accessibility features or accommodations to students 
Clanfy directions 

Repeat directions 

Redirect a student to the test 

Human reader for the Mathematics Assessment 

Human reader for the English language artsiliteracy assessment 
Scribe - English language arts/literacy assessment 


oO hl & Oo 


Scribe - Mathematics Assessment 
4. If you served as a human reader (read text aloud) to a student and/or served as a scribe, 
please indicate if you read the following documents (select all that apply): 

™ Not Applicable 


[™ "Appendex B: Test Administration Protocol for the Read-Aloud Accommodation for English Language 
Arts/Literacy Assessments and for Accessibility Feature for Mathematics" 


[™ “Appendix C: Protocol for the Use of the Scribe Accommodation” 

[ “Appendix |: Audio and Human Reader Guidelines for the ELA/Literacy Assessments” 

[ “Appendix J: Audio and Human Reader Guidelines for the Mathematics Assessments” 
5. Did you receive training on administering accessibility features/accommodations from 
your school or district? 

© Yes 


© No 
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PARCC 2014 Field Test 


6. Do you have any recommendations on improving training and information available on 
PARCC's accessibility features and accommodations? 


7. If you administered an accommodation and you are interested in providing additional 
feedback to PARCC, please provide your email address and PARCC will follow-up with 
you: (e.g., jsmith@emailaddress.com) 


(Sample Items/Tutorials) 
1. Did the students in your session(s) practice with PARCC sample items prior to 
administration? 
© Yes 
© No 
© Not sure/Don't know 
2. Did the students in your session(s) complete the PARCC tutorial prior to administration? 
© Yes 
© No 
© Not sure/Don't know 
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PARCC 2014 Field Test 
(After Completion of the Assessment(s) 


The Test Coordinator Manual allows for local and/or state decision on what students are 
allowed to do after they have completed testing. 


In accordance with your state or local policy, please indicate which of the following 
options was given to students after the completion of testing. 


© Students sat quietly until the testing session had ended. 
© Students were dismissed after they completed the testing. 
© Students quietly used a recreational book or other allowable material until the testing session had ended. 


(Final Feedback) 


Please use the space below to provide any other recommendations (not already provided 
in the questions above) that may help PARCC improve its administration policies and 
procedures (e.g., administration instructions, security policies/procedures, and/or the role 
of the Test Administrator). 


= | 
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PARCC 2014 Field Test 


Thank you for taking the Spring 2014 PARCC Field Test Survey. Select "Done" to submit your responses or "Prev" to 
retum to the survey to edit your responses. 
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Appendix D — Test Administrator Survey for Paper-based Administration 


PARCC 2014 Field Test 


Thank you for administering the PARCC Field Test. Now, we want to ask you some questions about 
the assessment(s) you just administered. We will use your answers to help improve the test 
administration process. 


* Please indicate the mode of assessment administration: 
© Computer-based (or tablet) administration 


© Paper-and-pencil administration 


Page 1 
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1. Please identify the location where you administered the assessment: 


a 


‘ 
‘ 
‘a 
a 
‘ 
F 
- 
‘ 
“ 
i 
ze 
‘ 
P 
é 


* 2, Please enter the name of the LEA/District in which you administered the assessment. 
District Name: | 

* 3, Please enter the name of the school in which you administered the assessment 
School Code: 


School Name: | 
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PARCC 2014 Field Test 


4. Please identify the grade level(s)/course(s) in which you administered the assessment(s) 
(select all that apply). 
Abbreviation Definitions: 
ELAIL PBA = English Language Art/Literacy Performance-Based Assessment 
ELAI/L EOY = English Language Art/Literacy End of Year 
ELAIL PBA ELA/L EOY 
= co 


a Co 
Cc Cc 
Cc Cc 
rc a 
os rc 
Br a 
3 Cc 
a Ci 
Cc Cc 


aye 2£uU 
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5. Please identify the grade level(s)/course(s) in which you administered the assessment(s) 
(select all that apply). 


Math PBA = Math Performance-Based Assessment 
Math EOY = Math End of Year 
Math PBA 


Not Applicable - I did C 
not administer Math 


Algebra 1 
Geometry 
Algebra 2 
Integrated Math 1 
integrated Math 2 
integrated Math 3 


(Test Administrator's Training Experience) 


Lh CBU ate Oe Oat Osu oO 
CEO Oe Oe Oe oa oO 


1. Did you receive any training on administering the assessment(s) from your state, school 
or district? (select all that apply) 


™ Yes, | received training from my state. 
[ Yes, | received training from my district. 
™ Yes, | received training from my school. 
[ No, I did not receive any training from my state, district or school. 


(Test Administration Experience) 

Please rate your level of agreement with each of the following statements (1-4). 

1. The policies and procedures within the Test Administrator Manual were easy for me to 
understand. 


© Strongly Disagree © Disagree © Agree © Strongly Agree 
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PARCC 2014 Field Test 


2. The instructions (including scripts for administering the assessment) within the Test 
Administrator Manual were easy for me to implement. 


© Strongly Disagree © Disagree © Agree © Strongly Agree 
3. Students appeared to understand the instructions | read to them during test 
administration. 

c Strongly Dis. agree Cc Disagree Cc Agree Cc Strongly Agree 
4. The instructions | read to the students covered all of the information necessary to take 
the test. 

© Strongly Disagree © Disagree C Agree © Strongly Agree 
5. Please use the space below to describe any questions students asked about the test- 


taking instructions you read to them. 
— | 


ps | 


(Test Environment) 


1. Were there any problems in the physical environment during the test session that might 
have negatively affected student performance? 


© Yes 


© No 
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PARCC 2014 Field Test 


1a. Please indicate the nature of the problem in the physical environment. (select all that 
apply) 


™ Lighting was insufficient 
™ Other (please use the space below to identify ‘other’ issue) 


Other (please specify ) 
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PARCC 2014 Field Test 
(PARCC's Timing Policy) 
(PARCC’s Timing Policy) 


1. Indicate the proportion of students who needed the Additional Time Allowed. 
© None 


© Less than a third of students 
© Approximately half of students 
e 


More than half of students 
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PARCC 2014 Field Test 
(PARCC's Timing Policy) 
1a. Please indicate how much additional time was used by the majority of these students. 
© Less than 10 minutes 


© Between 10-20 minutes 
© More than 20 minutes 
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(Accessibility Features and Accommodations) 
1. Are you aware there is an approved PARCC Accessibility Features and 
Accommodations Manual? 


© Yes 


© No 
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PARCC 2014 Field Test 


1a. Did you read the “Accessibility Features and Accommodations Manual” prior to 
administration? 


© Yes 


© No 
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PARCC 2014 Field Test 


2. Indicate whether you administered any of the following accessibility features or 
accommodations to your students during the Field Test administration? (please select all 


that apply): 

| did not administer any accessibility features or accommodations to students 
Clanfy directions 

Repeat directions 

Redirect a student to the test 

Human reader for the Mathematics Assessment 

Human reader for the English language artsiliteracy assessment 
Scribe - English language arts/literacy assessment 


cr Gl EB Gite (fh Gc 


Scribe - Mathematics Assessment 
3. If you served as a human reader (read text aloud) to a student and/or served as a Scribe, 
please indicate if you read the following documents (please select all that apply): 

[ Not Applicable 


[™ “Appendix B: Test Administration Protocol for the Read-Aloud Accommodation for English Language 
Arts/Literacy Assessments and for Accessibility Feature for Mathematics" 


[ “Appendix C: Protocol for the Use of the Scribe Accommodation” 

[ “Appendix |: Audio and Human Reader Guidelines for the ELA/Literacy Assessments” 

™ “Appendix J: Audio and Human Reader Guidelines for the Mathematics Assessments” 
4. Did you receive training on administering accessibility features/accommodations from 
your school or district? 

© Yes 


© No 


Page 28 
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5. Do you have any recommendations on improving training and information available on 
PARCC's accessibility features and accommodations? 


* 


6. If you administered an accommodation and you are interested in providing additional 
feedback to PARCC, please provide your email address and PARCC will follow-up with 
= (e.g., jsmith@emailaddress.com): 


(Sample Items/Tutorials) 


1. Did the students in your session(s) have an opportunity to practice with PARCC sample 
items prior to administration? 

© Yes 

© No 

© Not sure/Don't know 
2. Did the students in your session(s) have an opportunity to complete the PARCC tutorial 
prior to administration? 

© Yes 

© No 

© Not sure/Don't know 


Updated February 17, 2015 Page D-12 


PARCC Quality of Test Administration 


PARCC 2014 Field Test 
(After Completion of the Assessment(s)) 


The Test Coordinator Manual allows for local and/or state decision on what students are 
allowed to do after they have completed testing. 


In accordance with your state or local policy, please indicate which of the following 
options was given to students after the completion of testing. 

© Students sat quietly until the testing session had ended. 

© Students were dismissed after they completed the testing. 

© Students quietly used a recreational book or other allowable material until the testing session had ended. 


(Final Feedback) 

Please use the space below to provide any other recommendations (not already provided 
in the questions above) that may help PARCC improve its administration policies and 
procedures (e.g., administration instructions, security policies/procedures, and/or the role 
of the Test Administrator). 


| 
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PARCC 2014 Field Test 
Thank You 


Thank you for taking the Spring 2014 PARCC Field Test Survey. Select "Done" to submit your responses or "Prev" to 
retum to the survey to edit your responses. 
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Appendix E — Student Survey Items (and Code Book) for 
Computer/Tablet-based PARCC Mathematics Field Test 


Now that you have finished the PARCC field test, we would like to know about your experience. We will 
use your responses to help make a better testing experience for students in the future. 


Directions 


Please answer the questions below as best that you can. If you do not understand a question, please ask 


your teacher or test administrator for help. 


1. Did you understand all of the directions read by the person who gave you the test? 
Variable Name = CBT_Math_Understand_lInstr 


Yes Value=1 
No Value =2 
2. How often was it hard to understand the directions for the questions on this test? 
Variable Name = CBT_Math_Difficult_Instr 
Almost Always Value =1 
Most of thetime Value =2 
Some of the time Value =3 
Almost Never Value = 4 
3. How many questions asked you about things you have not learned in school this year? 
Variable Name = CBT_Math_Qs_Not_Learned 
All of them Value =1 
Most of them Value = 2 
Few of them Value =3 
None of them Value = 4 
4. How difficult was this test? 
Variable Name = CBT_Math_DifTest 
It was easier than my school work Value =1 
It was about the same as my school work Value = 2 


It was harder than my school work Value = 3 
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5. Did you have enough time to finish this test? 
Variable Name = CBT_Math_Time_Finish 


| finished very early Value =1 
| finished on time Value = 2 
| had to rush to finish Value = 3 


| did not finish Value = 4 


6. How often do you use a computer or tablet at home? 
Variable Name = CBT_Math_Home_Tech 


Every day Value =1 
A few times a week Value =2 
A couple of times a month or less Value = 3 


| don’t have acomputer or atabletat home Value=4 


7. How often do you use a computer or tablet in school? 
Variable Name = CBT_Math_School_Tech 


Every day Value =1 
A few times a week Value = 2 
A couple of times a month or less Value = 3 


| have never used a computer or tablet inschool Value =4 


8. When writing a story or essay, how often do you use a computer or tablet? 
Variable Name = CBT_Math_Tech_Writing 


All of the time Value =1 
Most of the time Value = 2 
Some of the time Value = 3 


Never Value = 4 
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9. Would you rather take this test on paper OR on a computer or tablet? 
Variable Name = CBT_Math_Test_Pref 


On paper Value =1 


Onacomputer ortablet Value=2 


10. What did you use to take this test? 
Variable Name = CBT_Math_TechType 


Desktop computer Value =1 
Laptop computer Value = 2 
Tablet with a keyboard Value = 3 


Tablet without akeyboard Value=4 


11. How many times did you practice on a computer or tablet to get ready for this test? 
Variable Name = CBT_Math_TechPractice 


Never Value =1 
Once Value = 2 


More than once Value = 3 


12. Was it easy to use the highlighter 
Variable Name = CBT_Math_Highlight 


Yes, it was easy Value =1 
No, it was hard Value = 2 


| did not use the highlighter Value =3 


13. Was it easy to type your answers? 
Variable Name = CBT_Math_Type 


Yes, it was easy Value =1 
No, it was hard Value = 2 


| did not type any answers Value =3 
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14. 


15. 


16. 


17. 


18. 


Was it easy to make pictures or words bigger or smaller? 
Variable Name = CBT_Math_Change_ Size 


Yes, it was easy Value =1 
No, it was hard Value = 2 


| did not change the size of pictures or words Value=3 


Was it easy to enter math symbols and numbers for your answers? 
Variable Name = CBT_Math_Enter_SymNum 


Yes, it was easy Value =1 
No, it was hard Value = 2 
| did not enter math symbols or numbers Value = 3 


Was it easy to use the calculator? 
Variable Name = CBT_Math_Calc 


Yes, it was easy Value =1 
No, it was hard Value = 2 


| did not use the calculator Value=3 


Did any of the problems listed below happen during your test? (Select all that apply) 


The computer stopped working CBT_Math_Comp_Stop Value =1 

The computer worked slowly CBT_Math_Comp_Slow Value = 2 

| had a hard time dragging or moving things on the screen CBT_Math_Move_Objects Value =3 
| had a hard time making changes to my answers CBT_Math_Difficult_Change_Ans Value=4 


No problems happened during the test CBT_Math_No_Probs Value =5 


Please use the space below to tell us about anything that was confusing or unclear about the test. 
We are not asking whether a question on the test was hard, but we would like to know if there was 
anything about the test that didn’t make sense. CBT_Math_Feedback_txt 
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Appendix F — Student Survey Items (and Code Book) for 
Computer/Tablet-based PARCC ELA Field Test 


Now that you have finished the PARCC field test, we would like to know about your experience. We will 
use your responses to help make a better testing experience for students in the future. 


Directions 


Please answer the questions below as best that you can. If you do not understand a question, please ask 
your teacher or test administrator for help. 


1. Did you understand all of the directions read by the person who gave you the test? 
Variable Name = CBT_ELA_Understand_Instr 


Yes Value =1 


No Value = 2 


2. How often was it hard to understand the directions for the questions on this test? 
Variable Name = CBT_ELA_Difficult_Instr 


Almost Always Value =1 
Most of the time Value = 2 
Some of the time Value = 3 


Almost Never Value =4 


3. How many questions asked you about things you have not learned in school this year? 
Variable Name = CBT_ELA_Qs_Not_Learned 


All of them Value =1 
Most of them Value =2 
Few of them Value = 3 
None of them Value = 4 


4. How difficult was this test? 
Variable Name = CBT_ELA_DifTest 


It was easier than my school work Value =1 
It was about the same as my school work Value = 2 
It was harder than my school work Value = 3 


Updated February 17, 2015 Page F-1 


PARCC Quality of Test Administration 


5. Did you have enough time to finish this test? 
Variable Name = CBT_ELA_Time_Finish 
| finished very early Value =1 
| finished on time Value = 2 
| had to rush to finish Value = 3 


| did not finish Value = 4 


6. How often do you use a computer or tablet at home? 
Variable Name = CBT_ELA_Home_Tech 


Every day Value =1 
A few times a week Value = 2 
A couple of times a month or less Value = 3 


| don’t have a computer or a tablet at home Value =4 


7. How often do you use a computer or tablet in school? 
Variable Name = CBT_ELA_School_Tech 


Every day Value =1 
A few times a week Value = 2 
A couple of times a month or less Value = 3 
| have never used a computer or tablet in school Value = 4 


8. When writing a story or essay, how often do you use a computer or tablet? 
Variable Name = CBT_ELA_Tech_Writing 


All of the time Value =1 
Most of the time Value = 2 
Some of the time Value = 3 
Never Value = 4 


9. Would you rather take this test on paper OR on a computer or tablet? 
Variable Name = CBT_ELA_Test_Pref 


On paper Value =1 


On a computer or tablet Value = 2 
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10. 


11. 


12. 


13. 


14. 


What did you use to take this test? 
Variable Name = CBT_ELA_TechType 


Desktop computer Value =1 
Laptop computer Value = 2 
Tablet with a keyboard Value = 3 
Tablet without a keyboard Value = 4 


Quality of Test Administration 


How many times did you practice on a computer or tablet to get ready for this test? 


Variable Name = CBT_ELA_TechPractice 


Never Value =1 
Once Value = 2 
More than once Value = 3 


Was it easy to use the highlighter 
Variable Name = CBT_ELA_Highlight 


Yes, it was easy Value =1 
No, it was hard Value = 2 


| did not use the highlighter Value = 3 


Was it easy to type your answers? 
Variable Name = CBT_ELA_Type 


Yes, it was easy Value =1 
No, it was hard Value = 2 
| did not type any answers Value = 3 


Was it easy to make pictures or words bigger or smaller? 
Variable Name = CBT_ELA_Change_ Size 


Yes, it was easy Value =1 
No, it was hard Value = 2 
| did not change the size of pictures or words Value = 3 
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15. 


16. 


17. 


18. 


Was it easy to move back and forth between passages or stories? 
Variable Name = CBT_ELA_Change_Pass 


Yes, it was easy Value =1 
No, it was hard Value = 2 
| did not move between stories or passages Value = 3 


Was it easy to find information in the passages or stories when answering questions? 
Variable Name = CBT_ELA_Find_Info 


Yes, it was easy Value =1 
No, it was hard Value = 2 
| did not move between stories or passages Value = 3 


Did any of the problems listed below happen during your test? (Select all that apply) 


The computer stopped working CBT_ELA_Comp_Stop Value =1 
The computer worked slowly CBT_ELA_Comp_Slow Value = 2 
| had a hard time dragging or moving things on the screen 
CBT_ELA_Move_Objects Value = 3 
| had a hard time making changes to my answers CBT_ELA_Change_Ans Value=4 
No problems happened during the test CBT_ELA_No_Probs Value =5 


Please use the space below to tell us about anything that was confusing or unclear about the test. 
We are not asking whether a question on the test was hard, but we would like to know if there was 
anything about the test that didn’t make sense. CBT_ELA_Feedback_txt 
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Appendix G — Student Survey Items (and Code Book) for 
Paper-based PARCC Math Field Test 


Now that you have finished the PARCC field test, we would like to know about your experience. We will 
use your responses to help make a better testing experience for students in the future. 


Directions 


Please answer the questions below as best that you can. If you do not understand a question, please ask 
your teacher or test administrator for help. 


1. Did you understand all of the directions read by the person who gave you the test? 
Variable Name = PBT_Math_Understand_Instr 


Yes Value =1 
No Value = 2 


2. How often was it hard to understand the directions for the questions on this test? 
Variable Name = PBT_Math_Difficult_Instr 


Almost Always Value=1 
Most of the time Value = 2 
Some of the time Value = 3 
Almost Never Value = 4 


3. How many questions asked you about things you have not learned in school this year? 
Variable Name = PBT_Math_Qs_Not_Learned 


All of them Value =1 
Most of them Value = 2 
Few of them Value = 3 
None of them Value = 4 


4. How difficult was this test? 
Variable Name = PBT_Math_DifTest 


It was easier than my school work Value =1 
It was about the same as my school work Value = 2 


It was harder than my school work Value = 3 
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5. Did you have enough time to finish this test? 
Variable Name = PBT_Math_Time_Finish 


| finished very early Value =1 
| finished on time Value = 2 
| had to rush to finish Value = 3 
| did not finish Value = 4 


6. How often do you use a computer or tablet at home? 
Variable Name = PBT_Math_Home_Tech 


Every day Value =1 
A few times a week Value = 2 
A couple of times a month or less Value = 3 


| don’t have a computer or a tablet at home Value = 4 


7. How often do you use a computer or tablet in school? 
Variable Name = PBT_Math_School_Tech 


Every day Value = 1 
A few times a week Value = 2 
A couple of times a month or less Value = 3 
| have never used a computer or tablet in school Value = 4 


8. When writing a story or essay, how often do you use a computer or tablet? 
Variable Name = PBT_Math_Tech_Writing 


All of the time Value =1 
Most of the time Value = 2 
Some of the time Value = 3 
Never Value = 4 
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9. Would you rather take this test on paper OR on a computer or tablet? 
Variable Name = PBT_Math_Test_Pref 


On paper Value =1 


On a computer or tablet Value = 2 
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Appendix H — Student Survey Items (and Code Book) for 
Paper-based PARCC ELA Field Test 


Now that you have finished the PARCC field test, we would like to know about your experience. We will 
use your responses to help make a better testing experience for students in the future. 


Directions 


Please answer the questions below as best that you can. If you do not understand a question, please ask 
your teacher or test administrator for help. 


1. Did you understand all of the directions read by the person who gave you the test? 
Variable Name = PBT_ELA_Understand_Instr 


Yes Value =1 


No Value = 2 


2. How often was it hard to understand the directions for the questions on this test? 
Variable Name = PBT_ELA_Difficult_Instr 


Almost Always Value =1 
Most of the time Value = 2 
Some of the time Value = 3 


Almost Never Value = 4 


3. How many questions asked you about things you have not learned in school this year? 
Variable Name = PBT_ELA_Qs_Not_Learned 


All of them Value =1 
Most of them Value = 2 
Few of them Value = 3 


None of them Value = 4 
4. How difficult was this test? 
Variable Name = PBT_ELA_DifTest 


It was easier than my school work Value =1 
It was about the same as my school work Value = 2 


It was harder than my school work Value = 3 
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5. Did you have enough time to finish this test? 
Variable Name = PBT_ELA_Time_Finish 


| finished very early Value =1 
| finished on time Value =2 
| had to rush to finish Value = 3 
| did not finish Value = 4 


6. How often do you use a computer or tablet at home? 
Variable Name = PBT_ELA_Home_Tech 


Every day Value =1 
A few times a week Value = 2 
A couple of times a month or less Value = 3 
| don’t have a computer or a tablet at home Value =4 


7. How often do you use a computer or tablet in school? 
Variable Name = PBT_ELA_School_Tech 


Every day Value =1 
A few times a week Value = 2 
A couple of times a month or less Value = 3 


| have never used a computer or tablet inschool Value =4 
8. When writing a story or essay, how often do you use a computer or tablet? 
Variable Name = PBT_ELA_Tech_Writing 
All of the time Value =1 
Most of the time Value = 2 
Some of the time Value = 3 


Never Value = 4 
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9. Would you rather take this test on paper OR on a computer or tablet? 
Variable Name = PBT_ELA_Test_Pref 


On paper Value =1 
Onacomputer ortablet Value=2 
10. Was it easy to flip back and forth between pages to find the story or passage that belonged to a 


question? 
Variable Name = PBT_ELA_Change_Pass 


Yes, it was easy Value =1 
No, it was hard Value = 2 


| did not flip back and forth between pages to find a passage Value = 3 
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Appendix | — Item-level Results from Test Administrator Survey: 
Computer/Tablet-based Administration 


Administration Information by State 


Table 1. TA Survey: Q1. Please indicate the mode of assessment administration: 
Computer-based (or tablet) Administration Paper-and-pencil Administration 


State n % n % 

Arizona (N = 493) 295 59.8 198 40.2 
Arkansas (N = 471) 314 67.0 157 33.0 
Colorado (N = 223) 152 68.2 71 31.8 
District of Columbia (D.C.; N = 35) 31 89.0 4 11.0 
Florida (N = 8) 7 87.5 1 12.5 
Illinois (N = 1024) 737 72.0 287 28.0 
Louisiana (N = 311) 304 98.0 7 2.0 
Maryland (N = 1171) 710 60.6 461 39.4 
Massachusetts (N = 776) 601 774 175 22.6 
Mississippi (N = 148) 94 63.5 54 36.5 
New Jersey (N = 788) 785 99.6 3 0.4 
New Mexico (N = 48) 48 100.0 0 0.0 
New York (N = 57) 54 94.7 3 5.3 
Ohio (N = 393) 240 61.1 153 38.9 
Rhode Island (N = 183) 128 69.9 55 30.1 
Tennessee (N = 183) 128 69.9 55 30.1 
All States (N = 6322) 4635° 73.3 1687° 26.7 


Note. Percentages are based on the total number of valid responses for each state. Number of missing for computer- 
based administration = 7. Number of missing for paper-and-pencil administration = 3. 

“Includes the 7 cases that did not indicate their state. 

° Includes the 3 cases that did not indicate their state. 
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Background Information 


Table 2. TA Survey: Q4a-b. Please identify the grade level(s)/course(s) in which you administered the assessment(s) 
(select all that apply)’. 


Not Applicable 
ELA/L Performance-based ELA/L End-of-Year Math Performance- Math End-of-Year 
Assessment Assessment based Assessment Assessment 

State n % n % n % n % 
Arizona Sih 19.3 47 15.9 49 16.6 43 14.6 
Arkansas 65 20.7 65 20.7 86 27.4 59 18.8 
Colorado 28 18.4 27 17.8 33 21.7 24 15.8 
D.C. 8 25.8 6 19.4 8 25.8 8 25.8 
Florida 1 14.3 1 14.3 1 14.3 1 14.3 
Illinois 163 22.1 155 21.0 195 26.5 163 22.1 
Louisiana 62 20.4 56 18.4 67 22.0 37 12.2 
Maryland 124 17.5 112 15.8 193 212 100 14.1 
Massachusetts 113 18.8 103 17.1 120 20.0 81 13.5 
Mississippi 27 28.7 13 13.8 19 20.2 21 22.3 
New Jersey 148 18.9 123 15.7 210 26.8 134 17.1 
New Mexico 12 25.0 10 20.8 7 14.6 6 12.5 
New York 14 25.9 11 20.4 10 18.5 11 20.4 
Ohio 34 14.2 52 21.7 52 21.7 43 17.9 
Rhode Island 28 21.9 29 22.7 27 21.1 20 15.6 
Tennessee 23 18.0 25 19.5 31 24.2 23 18.0 
All States 907 19.6 835 18.0 1111 24.0 774 16.7 


Note. * Percentages reported for “select all that apply” items should be interpreted with caution, as it is not possible 
to differentiate “missing” (i.e., skipped) vs. “not selected” responses. 
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Table 3. TA Survey: Q4a. Please identify the grade level(s)/course(s) in which you administered the assessment(s) (select all that apply“). 


ELA/L Performance-Based Assessment 


Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10 Grade 11 
State n % n % n % n % n % n % n % n % n % 
Arizona (n = 238) 16 6.7 20 8.4 15 6.3 26 10.9 20 8.4 23 9.7 6 2.5 15 6.3 15 6.3 
Arkansas (n = 249) 4l 16.5 20 8.0 22 8.8 17 6.8 17 6.8 10 4.0 12 4.8 12 4.8 9 3.6 
Colorado (n = 124) 20 16.1 30. =. 24.2 17 13.7 6 4.8 10 8.1 7 5.6 5 4.0 4 3.2 2 1.6 
D.C. (n = 23) 2 8.7 2 8.7 0 0.0 3 13.0 3 13.0 2 8.7 0 0.0 0 0.0 0 0.0 
Florida (n = 6) 0 0.0 2 33.3 0 0.0 1 16.7 1 16.7 0 0.0 0 0.0 0 0.0 0 0.0 
Illinois (n = 574) 64 11.1 46 8.0 77 13.4 52 9.1 37 6.4 47 8.2 31 54 37 64 27 4.7 
Louisiana (n = 242) 30 12.4 26 10.7. 925 10.3 27) =11.2 19 79 9 3.7 8 3.3 15 6.2 13 5.4 
Maryland (n = 586) 135 23.0 111 189 3 0.5 33 5.6 71 12.1 8 1.4 52 8.9 11 19 70 11.9 
Massachusetts (n= 488) 53 10.9 47 9.6 41 8.4 51 10.5 55 11.3 37 7.6 34 7.0 4 0.8 27 5.5 
Mississippi (n = 67) 0 0.0 4 6.0 2 3.0 6 9.0 0 0.0 1 1.5 10 14.9 5 7.5 7 10.4 
New Jersey (n = 637) 56 8.8 61 9.6 68 10.7 31 4.9 54 8.5 45 7.1 41 6.4 36 5.7 38 6.0 
New Mexico (n = 36) 2 5.6 0 0.0 1 2.8 3 8.3 0 0.0 4 11.1 1 2.8 0 0.0 1 2.8 
New York (n = 40) 2 5.0 0 0.0 1 2.5 2 5.0 3 7.5 2 5.0 2 5.0 0 0.0 1 2:5 
Ohio (n = 206) 12 5.8 18 8.7 28 13.6 11 5.3 13 6.3 18 8.7 12 5.8 10 4.9 > 2.4 
Rhode Island(n=100) 16 16.0 4 4.0 17 17.0 7 7.0 9 9.0 4 4.0 3 3.0 0 0.0 6 6.0 
Tennessee (n = 105) 7 6.7 19 18.1 10 9.5 14 13.3 14 13.3 18 17.1 3 2.9 7 6.7 6 5.7 
All States (n = 3728) 456 12.2 411 11.0 328 8.8 292 7.8 326 8.7 236 63 220 35.9 156 4.2 227 6.1 


Note. Individuals who selected N/A for this item (see Table 2) have been excluded from analyses. 
“ Percentages reported for “select all that apply” items should be interpreted with caution, as it is not possible to differentiate “missing” (i.e., skipped) vs. “not 


selected” responses. 
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Table 4. TA Survey: Q4a. Please identify the grade level(s)/course(s) in which you administered the assessment(s) (select all that apply“). 
ELA/L End-of- Year Assessment 


Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10 Grade 11 
State n % n % n % n % n % n % n % n % n % 
Arizona (n = 248) 11 4.4 15 6.0 15 6.0 20 8.1 13 5.2 7 2.8 8 3.2 5 2.0 7 2.8 
Arkansas (n = 249) 16 6.4 11 44 14 5.6 13 5.2 11 44 4 1.6 12 48 15 6.0 8 3.2 
Colorado (n = 125) 7 5.6 9 7.2 4 3.2 4 3.2 5 4.0 2 1.6 1 0.8 l 0.8 2 1.6 
D.C. (n = 25) 4 16.0 2 8.0 5 20.0 1 4.0 1 4.0 1 4.0 0 0.0 0 0.0 0 0.0 
Florida (n = 6) 0 0.0 0 0.0 0 0.0 1 16.7 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 
Illinois (n = 582) 27 4.6 23 4.0 22 3.8 29 5.0 25 43 22 3.8 19 3.3 26 4.5 11 1.9 
Louisiana (n = 248) 6 2.4 5 2.0 7 2.8 4 1.6 3 1.2 4 1.6 4 1.6 l 0.4 l 0.4 
Maryland (n = 598) 24 4.0 31 5.2 5 0.8 8 1.3 18 3.0 3 0.5 12 2.0 3 0.5 11 1.8 
Massachusetts (n= 498) 20 4.0 24 4.8 19 3.8 12 24 18 3.6 17 3.4 9 1.8 7 1.4 8 1.6 
Mississippi (n = 81) 0 0.0 6 7.4 1 1.2 3 37 0 0.0 2 25 14 17.3 14 17.3 0 0.0 
New Jersey (n = 662) 16 2.4 15 2.3 20 3.0 19 2.9 15 2.3 8 1.2 12 18 19 2.9 13 2.0 
New Mexico (n = 38) 2 5.3 1 2.6 2 5.3 2 5.3 4 10.5 0 0.0 2 5.3 5 13.2 2 5.3 
New York (n= 43) 7 16.3 0 0.0 4 9.3 6 14.0 1 2.3 1 2.3 1 2.3 1 2.3 1 2.3 
Ohio (n = 188) 3 1.6 4 2.1 16 8.5 5 2.7 7 3.7 10 5.3 7 3.7 4 2.1 7 3.7 
Rhode Island (n = 99) 4 4.0 1 1.0 7 7.1 3 3.0 1 1.0 0 0.0 1 1.0 2 2.0 1 1.0 
Tennessee (n = 103) 3 2.9 2 1.9 3 2.9 2 1.9 4 3.9 4 3.9 1 1.0 3 2.9 0 0.0 


All States (n = 3800) 150 3.9 149 3.9 144 3.8 132 3.5 126 3.3. 85 2.22103 2.7 106 28 72 1.9 
Note. Individuals who selected N/A for this item (see Table 2) have been excluded from analyses. 

“ Percentages reported for “‘select all that apply” items should be interpreted with caution, as it is not possible to differentiate “missing” (i.e., skipped) vs. “not 
selected” responses. 
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Table 5. TA Survey: Q4b. Please identify the grade level(s)/course(s) in which you administered the assessment(s) (select all that apply“). 


Math Performance-Based Assessment 


Grade3_ Grade4 Grade5  Grade6 Grade7 Grade8 ALGI GEO ALG 2 INT 1 INT 2 INT 3 

State n %® n % n % n % nn % nn % nn %® nn % nn % nn % Dn % TH GU 

Arizona (n = 246) 17 69 20 81 13 53 16 65 19 77 21 85 11 45 5 20 4 16 0 00 0 00 O 0.0 
Arkansas (n = 228) 6 26 28 123 18 7.9 13 57 7 31 12 53 #5 22 12 53 12 53 0 00 0 00 O 0.0 
Colorado (n = 119) 10 84 13 109 6 50 9 76 7 59 4 34 0 00 10 84 5 42 2 1.7 2 #17 #2 =«127 
D.C. (n= 23) 1 43 1 43 0 00 0 00 1 43 0 00 0 00 0 00 0 00 0 00 0 00 0. 0.0 
Florida (n = 6) 1 16.7 4 667 0 00 0 00 1 167 0 00 0 00 0 00 0 00 0 00 0 00 OO 0.0 
Illinois (n = 542) 45 83 54 100 32 59 31 5.7 26 48 37 68 36 66 27 50 27 50 6 11 1 #02 #1 O02 
Louisiana (n = 237) 23 9.7 27 114 21 89 15 63 11 46 10 42 8 34 6 25 13 55 0 00 0 00 0 0.0 
Maryland (n = 517) 59 114 81 15.7 2 04 72 13.9 59 114 6 12 4 O8 47 9.1 25 48 #%J #02 | O02 1 O02 
Massachusetts (n=481) 41 8.5 58 12.1 32 67 33 69 29 60 25 52 15 3.1 11 23 21 44 14 29 1 O02 21 44 
Mississippi (n = 75) 4 53 19 253 7 93 9 120 0 00 3 40 0 00 1 13 1 13 0 00 0 00 O0O 0.0 
New Jersey (n=575) 39 68 36 63 18 3.1 20 35 30 52 28 49 29 50 34 59 26 45 2 03 2 03 2 = 03 
New Mexico (n = 41) 3. 73 2 49 1 24 1 24 2 49 1 24 2 49 0 00 0 00 0 00 1 24 0. 0.0 
New York (n= 44) 245 2 45 2 45 1 23 2 45 0 00 1 23 4 91 3 68 0 00 0 00 0 0.0 
Ohio (n = 188) 10 53 11 59 14 74 12 64 10 53 12 64 8 43 3 #416 10 53 11 59 4 21 #5 2.7 
Rhode Island(n=101) 12 119 3 30 3 30 9 89 13 129 4 40 2 20 7 69 5 50 0 00 1 #10 #O 0.0 
Tennessee (n = 97) 13 134 4 41 #14 144 12 124 15 155 8 82 4 41 2 21 5 52 0 00 0 00 O 0.0 
All States (n= 3524) 286 8.1 363 10.3 183 5.2 254 7.2 232 66 171 49 125 35 169 48 158 45 36 10 13 04 32 09 


Note. Individuals who selected N/A for this item (see Table 2) have been excluded from analyses. 


“ Percentages reported for “‘select all that apply” items should be interpreted with caution, as it is not possible to differentiate “missing” (i.e., skipped) vs. “not 


selected” responses. 
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Table 6. TA Survey: Q4b. Please identify the grade level(s)/course(s) in which you administered the assessment(s) (select all that apply“). 


Math End-of-Year Assessment 


Grade3  Grade4 Grade5  Grade6 Grade7 Grade8 ALGI GEO ALG 2 INT 1 INT 2 INT 3 

State n %® n % n % n % n % n WM nn Ww Dn Mw Hn % 1n wy Hn WwW Tn GU 

Arizona (n = 252) 14 56 9 36 20 79 6 24 12 48 14 56 4 16 8 32 6 24 0 00 0 00 O 0.0 
Arkansas (n = 255) 10 39 18 7.1 12 47 8 3.1 #2 #08 11 43 17 67 9 35 8 3.1 0 00 0 00 O 0.0 
Colorado (n = 128) 4 31 4 31 4 31 3 23 7 55 #41 08 O 00 4 31 0 00 0 00 0 00 0 00 
D.C. (n = 23) 1 43 3 130 0 00 5 21.7 0 00 0 00 1 43 1 43 0 00 4 174 0 00 O 0.0 
Florida (n = 6) 0 00 1 167 0 00 0 00 0 00 0 00 0 00 0 00 0 00 0 00 0 00 OO 0.0 
Illinois (n = 574) 34 59 44 7.7 22 38 21 3.7 24 42 10 1.7 14 #24 22 38 13 23 4 07 #1 02 #2 «03 
Louisiana (n = 267) 8 30 12 45 5 19 6 22 6 22 8 30 2 07 5 19 4 15 0 00 0 00 O- 0.0 
Maryland (n= 610) 23 3.8 25 41 2 O03 17 28 18 30 6 10 2 03 9 15 #7 #11 #1 #02 #41 «02 1 0.2 
Massachusetts (n= 520) 24 46 25 48 29 56 23 44 19 3.7 15 29 12 23 4 O08 12 23 1 #02 0 00 2 O04 
Mississippi (n = 73) 0 00 2 27 0 00 1 14 0 00 4 55 1 14 0 00 0 00 0 00 0 00 O 0.0 
New Jersey (n = 651) 21 32 15 23 15 23 15 23 29 45 17 26 31 #48 #7 #41 11 =#17 #0 00 0 00 O- 0.0 
New Mexico (n = 42) 2 48 1 24 3 71 0 00 4 95 1 24 4 95 0 00 0 00 1 24 0 00 O 0.0 
New York (n = 43) 2 47 #6 140 2 47 0 00 2 47 #0 00 7 163 0 00 5 116 0 00 0 00 O- 0.0 
Ohio (n = 197) 10 5.1 3 #15 10 51 3 #15 #3 #15 #2 #10 12 61 2 10 8 41 21 #05 #1 05 1 0.5 
Rhode Island (n = 108) 9 83 0 00 4 37 2 19 3 28 0 00 6 56 5 46 5 46 0 00 0 00 OO 0.0 
Tennessee (n = 105) 1 10 0 00 2 19 2 19 1 10 3 29 2 19 0 00 4 38 0 00 0 00 O 0.0 
All States (n = 3861) 163 4.2 168 44 130 3.4 112 2.9 130 34 92 24 115 30 76 20 83 2.1 12 03 3 O1 6 O02 


Note. Individuals who selected N/A for this item (see Table 2) have been excluded from analyses. 


“ Percentages reported for “‘select all that apply” items should be interpreted with caution, as it is not possible to differentiate “missing” (i.e., skipped) vs. “not 


selected” responses. 
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Test Administrator’s Training Experience 


Table 7. TA Survey: Q1. Have you ever administered a computer-based test to students before (either at the state or 
district level)? 


Yes No 

State n % n % 

Arizona 152 52.2 139 47.8 
Arkansas 117 37.5 195 62.5 
Colorado 100 66.7 50 33.3 
D.C. 16 51.6 15 48.4 
Florida 3 42.9 4 57.1 
Illinois 429 58.4 305 41.6 
Louisiana 158 52.5 143 47.5 
Maryland 474 66.9 235 33.1 
Massachusetts 124 20.7 475 79.3 
Mississipp1 38 40.9 55 59.1 
New Jersey 186 23.7 599 76.3 
New Mexico 36 75.0 12 25.0 
New York 27 50.0 27 50.0 
Ohio 99 41.4 140 58.6 
Rhode Island 68 53.1 60 46.9 
Tennessee 100 78.1 28 21.9 
All States 2129 46.1 2486 53.9 


Note. Numbers and percentages are based on the total number of valid responses (missing = 20). 


Table 8. TA Survey: Q2. Did you complete any of the online PARCC training modules (as covered in the online 
training found here https://parcc.tms.pearson.com/)? 


Yes No 

State n % n % 

Arizona 215 73.6 77 26.4 
Arkansas 271 86.6 42 13.4 
Colorado 109 71.7 43 28.3 
D.C. 21 67.7 10 32.3 
Florida 5 71.4 2 28.6 
Illinois 521 71.3 210 28.7 
Louisiana 231 76.2 72 23.8 
Maryland 422 59.7 285 40.3 
Massachusetts 452 75.8 144 24.2 
Mississipp1 71 76.3 22 23.7 
New Jersey 643 82.2 139 17.7 
New Mexico 35 72.9 13 27.1 
New York 41 75.9 13 24.1 
Ohio 172 72.3 66 27.7 
Rhode Island 106 82.8 22 17.2 
Tennessee 75 58.6 53 41.4 
All States 3396 73.7 1213 26.3 


Note. Numbers and percentages are based on the total number of valid responses (missing = 26). 
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Table 9. TA Survey: Q2a. Please indicate which of the online training models you completed (select all that apply“). 


Accessibility Sutin Tan Emerging 
Features and Computer-based a? Technologies and 
. Paper-based Test : Infrastructure Student Data : : 
Accommodations moh i Testing Test . Technology Setup . Security with 
. Administration fee ; Trial/Dress Upload Overview 
with Computer- Administration Computer-based 
: Rehearsal f 
Based Testing Testing 
State n % n % n % n % n % n % n % 
Arizona (n = 218) 85 39.0 9 4.1 158 72.5 47 21.6 34 15.6 39 17.9 19 8.7 
Arkansas (n = 272) 118 43.4 10 3.7 234 86.0 54 19.9 32 11.8 31 11.4 26 9.6 
Colorado (n= 109) 54 49.5 > 4.6 88 80.7 9 8.3 20 18.3 21 19.3 12 11.0 
D.C. (n= 21) 10 47.6 2 9.5 20 95.2 2 9.5 6 28.6 5 23.8 2 9.5 
Florida (n = 5) 1 20.0 0 0.0 0 0.0 1 20.0 0 0.0 0 0.0 0 0.0 
Illinois (n = 527) 192 36.4 27 5.1 414 78.6 66 12.5 69 13.1 59 11.2 32 6.1 
Louisiana (n = 232) 101 43.5 8 3.4 189 81.5 29 12.5 35 15.1 26 11.2 22 9.5 
Maryland (n = 425) 177 41.6 7 1.6 293 68.9 65 15.3 40 9.4 24 5.6 16 3.8 
Massachusetts (n= 457) 194 42.5 8 1.8 369 80.7 61 13.3 56 12.3 48 10.5 38 8.3 
Mississippi (n = 72) 18 25.0 4 5.6 58 80.6 7 9.7 7 9.7 5 6.9 2 2.8 
New Jersey (n = 646) 250 38.7 21 3.3 503 77.9 144 22.3 87 13.5 85 13.2 50 7.7 
New Mexico (n = 35) 12 34.3 0 0.0 23 65.7 9 25:1 6 17.1 7 20.0 5 14.3 
New York (n= 41) 14 34.1 2 4.9 32 78.0 12 29.3 10 24.4 11 26.8 3 7.3 
Ohio (n = 174) 77 44.3 7 4.0 138 79.3 29 16.7 28 16.1 31 17.8 12 6.9 
Rhode Island (n = 106) 53 50.0 1 0.9 80 15:5 11 10.4 7 6.6 13 12.3 6 oF 
Tennessee (n = 75) 25 33.3 2 277, 61 81.3 4 5.3 7 9.3 6 8.0 6 8.0 
All States (n = 3422) 1385 40.5 113 3.3 2670 78.0 551 16.1 445 13.0 414 12.1 252 7.4 


Note. Individuals who selected ‘No’ for item 2 (see Table 8) have been excluded from analyses. Individuals with missing responses to item 2 (see note in Table 
8) who responded to item 2a were included. 

“ Percentages reported for “‘select all that apply” items should be interpreted with caution, as it is not possible to differentiate “missing” (i.e., skipped) vs. “not 
selected” responses. 
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Table 10. TA Survey: Q2b. The PARCC online training(s) effectively informed me of the protocol to follow when 
administering the assessment. 


Strongl ; 

Not Applicable coheed Disneree Agee Strongly Agree Mean SD 
State n % n % n % n % n % 
Arizona 16 7.8 12 5.8 38 18.4 124 60.2 16 7.8 2.76 0.69 
Arkansas 11 41 10 3.7 39 146 190 71.2 17 64 2.84 0.59 
Colorado 6 5.8 6 5.8 16 15.4 73 70.2 3 29 2.74 0.61 
D.C. 2 10.5 0 0.0 2 10.5 15 78.9 0 0.0 2.88 0.33 
Florida 0 0.0 0 0.0 1 20.0 4 80.0 0 0.0 2.80 0.45 
Illinois 27 5.3 20 3.9 79 15.6 357 70.4 24 47 2.80 0.59 
Louisiana 10 44 11 49 27 12.0 151 67.1 26 11.6 2.89 0.66 
Maryland 45 11.3 22 5.5 51 12.8 259 65.1 21 5.3. 2.79 0.64 
Massachusetts 22 5.0 23 5.2 98 22.1 286 64.6 14 3.2 2.69 0.62 
Mississippi 5 7.0 4 5.6 14 19.7 42 59.2 6 8.5 2.76 0.70 
New Jersey 28 44 46 7.3 135 21.4 389 61.7 32 5.1 2.68 0.69 
New Mexico 4 12.1 0 0.0 8 24.2 21 63.6 0 0.0 2.72 0.45 
New York 2 49 1 2.4 6 14.6 28 68.3 4 98 2.90 0.60 
Ohio 12 71 15 8.8 360 21.2 =©98 = 557.6 9 5.3 2.64 0.73 
Rhode Island 5 5.0 5 5.0 24 238 59 584 8 79 2.73 0.69 
Tennessee 3 4.3 7 10.0 11 15.7 44 62.9 5 7.1 2.70 0.76 
All States 199 6.0 183 5.6 586 17.8 2144 65.0 185 5.6 2.75 0.65 


Note. Numbers and percentages are based on the total number of valid responses (missing = 125). Individuals who 
selected ‘No’ for item 2 (see Table 8) have been excluded from analyses. Individuals with missing responses to item 
2 (see note in Table 8) who responded to item 2b were included. Means and standard deviations were computed 
based on a 4-point scale (where Strongly Disagree = | and Strongly Agree = 4) with NA recoded as missing. 
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Table 11. TA Survey: Q2c. The PARCC online training(s) prepared me to resolve basic problems related to 
technology (e.g., logging students in, exiting the test, etc.). 


Strongl . Strong] 

Not Applicable eet Diss Petes fees Mean SD 
State n % n % n % n % n % 
Arizona 23 11.2 24 11.7 59 —- 28.6 91 44.2 9 44 246 0.78 
Arkansas 16 6.0 23 8.6 91 934.1 130 = 48.7 7 2.6 2.48 0.70 
Colorado 9 8.6 17 16.2 34 = 32.4 41 39.0 4 3.8 2.33 0.82 
D.C. 2 10.5 2 10.5 3 15.8 12 63.2 0 0.0 2.59 0.71 
Florida 1 20.0 1 20.0 1 20.0 1 20.0 1 20.00 2.50 1.29 
Illinois 55 10.9 64 12.7 148 29.4 227 45.0 10 2.0 2.41 0.76 
Louisiana 18 8.0 18 8.0 47 21.0 126 56.3 15 6.7 2.67 0.74 
Maryland 51 12.8 53 13.4 98 24.7 183 46.1 12 3.0 2.45 0.79 
Massachusetts 32 7.3 64 145 131 29.8 206 46.8 7 16 2.38 0.77 
Mississippi 6 8.6 4 5.7 20 =28.6 34 = 48.6 6 8.6 2.66 0.74 
New Jersey 40 64 97 15.4 205 32.6 270 42.9 17 2.7 2.35 0.78 
New Mexico 5 14.7 5 14.7 11 32.4 13 38.2 0 0.0 2.28 0.75 
New York 2 4.9 6 14.6 10 24.4 22 53.7 1 2.4 2.46 0.79 
Ohio 19 11.1 39 22.8 39 - 22.8 71 41.5 3 1.8 2.25 0.86 
Rhode Island 7 7A 18 18.2 39 39.4 33 33.3 2 2.0 2.21 0.78 
Tennessee 4 5.8 15 21.7 12 174 34 = 49.3 4 5.8 2.42 0.92 
All States 290 8.8 452 13.8 949 289 1498 45.6 98 3.0 241 0.78 


Note. Numbers and percentages are based on the total number of valid responses (missing = 135). Individuals who 
selected ‘No’ for item 2 (see Table 8) have been excluded from analyses. Individuals with missing responses to 

item 2 (see note in Table 8) who responded to item 2c were included. Means and standard deviations were computed 
based on a 4-point scale (where Strongly Disagree = | and Strongly Agree = 4) with NA recoded as missing. 


Table 12. TA Survey: Q3. Did you receive any other training, aside from the online 
PARCC training modules?(select all that apply °): 


No, I did not receive any other training. 


State n % 

Arizona 72 24.4 
Arkansas 38 12.1 
Colorado 12 7.9 
D.C. 6 19.4 
Florida 2 28.6 
Illinois 215 29.2 
Louisiana 28 9.2 
Maryland 82 11.5 
Massachusetts 150 25.0 
Mississippi 16 17.0 
New Jersey 106 13.5 
New Mexico 13 27.1 
New York 23 42.6 
Ohio 74 30.8 
Rhode Island 32 25.0 
Tennessee 30 23.4 
All States 899 19.4 


Note. * Percentages reported for “select all that apply” items should be interpreted with caution, as it is not possible to 
differentiate “missing” (i.e., skipped) vs. “not selected” responses. 
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Table 13. TA Survey: Q3. Did you receive any other training, aside from the online PARCC training 
modules ?(select all that apply“): 


Training from State Training from District Training from School 
State n % n % n % 
Arizona (n = 223) 11 4.9 111 49.8 105 47.1 
Arkansas (n = 276) 26 9.4 157 56.9 134 48.6 
Colorado (n = 140) 18 12.9 72 51.4 64 45.7 
D.C. (n = 25) 0 0.0 4 16.0 17 68.0 
Florida (n = 5) 0 0.0 1 20.0 4 80.0 
Illinois (n = 522) 28 5.4 295 56.5 225 43.1 
Louisiana (n = 276) 31 11.2 89 32.2 187 67.8 
Maryland (n = 628) 4 0.6 296 47.1 379 60.4 
Massachusetts (n = 451) 32 7.1 160 35.5 298 66.1 
Mississippi (n = 78) 2 2.6 22 28.2 59 75.6 
New Jersey (n = 679) 24 3.5 372 54.8 366 53.9 
New Mexico (n = 35) 2 5.7 20 57.1 14 40.0 
New York (n= 31) 5 16.1 11 35.5 15 48.4 
Ohio (n = 166) 12 7.2 94 56.6 61 36.7 
Rhode Island (n = 96) 21 21.9 30 31.3 58 60.4 
Tennessee (n = 98) 4 4.1 36 36.7 59 60.2 
All States (n = 3736) 220 5.9 1773 47.5 2049 54.8 


Note. Individuals who selected ‘No other training’ for this item (see Table 12) have been excluded from analyses. 
“ Percentages reported for “select all that apply” items should be interpreted with caution, as it is not possible to 
differentiate “missing” (i.e., skipped) vs. “not selected” responses. 


Test Administration Experience 


Table 14. TA Survey: Q1. The policies and procedures within the Test Administrator Manual were easy for me to 
understand. 


Strongly Disagree Disagree Agree Strongly Agree 
State n % n % n % n % 
Arizona 8 2.9 57 20.5 195 70.1 18 6.5 
Arkansas 24 8.1 71 23.9 186 63.6 13 4.4 
Colorado 7 4.7 46 30.9 92 61.7 4 237 
D.C. 0 0.0 2 74 23 85.2 2 7.4 
Florida 0 0.0 2 28.6 2 28.6 3 42.9 
Illinois 29 4.1 164 23.1 478 67.4 38 5.4 
Louisiana 14 4.9 43 15.1 195 68.7 32 11.3 
Maryland 54 7.9 135 19.7 449 65.5 47 6.9 
Massachusetts 18 3.2 143 25.2 382 67.3 25 4.4 
Mississippi 6 6.8 16 18.2 59 67.0 7 8.0 
New Jersey 45 6.1 202 27.3 459 61.9 35 4.7 
New Mexico 2 4.4 10 22.2 32 71.1 1 2.2 
New York 1 2.0 15 30.0 32 64.0 2 4.0 
Ohio 15 6.5 56 24.3 145 63.0 14 6.1 
Rhode Island 5 4.0 41 32.8 74 59.2 5 4.0 
Tennessee 9 7.3 28 22.8 74 60.2 12 9.8 
All States 238 5.4 1034 23.4 2882 65.3 258 5.8 


Note. Numbers and percentages are based on the total number of valid responses (missing = 223). 
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Table 15. TA Survey: Q2. The instructions (including scripts for administering the assessment) within the Test 
Administrator Manual were easy for me to implement. 


Strongly Disagree Disagree Agree Strongly Agree 
State n % n % n % n % 
Arizona 10 3.6 72 25.8 169 60.6 28 10.0 
Arkansas 32 10.8 96 32.3 160 53.9 9 3.0 
Colorado 24 16.3 50 34.0 71 48.3 2 1.4 
D.C. 0 0.0 7 25.9 19 70.4 1 3.7 
Florida 1 14.3 3 42.9 2 28.6 1 14.3 
Illinois 61 8.7 188 26.7 415 58.9 40 5.7 
Louisiana 21 74 63 22.2 162 57.0 38 13.4 
Maryland 71 10.3 198 28.8 372 54.1 46 6.7 
Massachusetts 28 4.9 138 24.3 370 65.1 32 5.6 
Mississipp1 6 6.7 19 21.3 55 61.8 9 10.1 
New Jersey 63 8.5 246 33.1 399 53.7 35 4.7 
New Mexico 3 6.5 16 34.8 26 56.5 1 2.2 
New York 2 4.2 11 22.9 32 66.7 3 6.3 
Ohio 21 9.1 61 26.5 135 58.7 13 5.7 
Rhode Island 20 16.0 44 35.2 56 44.8 5 4.0 
Tennessee 11 8.9 30 24.4 71 57.7 11 8.9 
All States 375 8.5 1245 28.2 2516 57.1 274 6.2 


Note. Numbers and percentages are based on the total number of valid responses (missing = 225). 


Table 16. TA Survey: Q3. Students appeared to understand the instructions I read to them during test 
administration. 


Strongly Disagree Disagree Agree Strongly Agree 
State n % n % n % n % 
Arizona 14 5.0 61 21.8 186 66.4 19 6.8 
Arkansas 18 6.1 78 26.4 188 63.5 12 4.1 
Colorado 14 9.5 37 25.2 93 63.3 3 2.0 
D.C. 3 11.5 7 26.9 15 a1) 1 3.8 
Florida 0 0.0 2 28.6 4 57.1 1 14.3 
Illinois 43 6.1 138 19.6 486 68.9 38 5.4 
Louisiana 7 2.5 54 18.9 183 64.2 41 14.4 
Maryland 32 4.7 136 19.8 472 68.6 48 7.0 
Massachusetts 23 4.1 115 20.4 403 71.3 24 4.2 
Mississippi 5 5.6 18 20.2 57 64.0 9 10.1 
New Jersey 47 6.3 149 20.1 511 68.9 35 4.7 
New Mexico 3 6.5 8 17.4 34 73.9 1 2.2 
New York 0 0.0 14 28.6 31 63.3 4 8.2 
Ohio 26 11.3 51 22.1 143 61.9 11 4.8 
Rhode Island 13 10.4 27 21.6 81 64.8 4 3.2 
Tennessee 7 5.7 24 19.5 79 64.2 13 10.6 
All States 256 5.8 921 20.9 2968 67.3 265 6.0 


Note. Numbers and percentages are based on the total number of valid responses (missing = 225). 
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Table 17. TA Survey: Q4. The instructions I read to the students covered all of the information necessary to take the 
test. 


Strongly Disagree Disagree Agree Strongly Agree 
State n % n % n % n % 
Arizona 9 32 60 21.7 187 67.5 21 7.6 
Arkansas 22 7.4 103 34.7 162 54.5 10 3.4 
Colorado 10 6.8 42 28.8 90 61.6 4 2.7 
D.C. 3 11.5 5 19.2 16 61.5 2 7.7 
Florida 0 0.0 2 28.6 4 57.1 1 14.3 
Illinois 54 TA 179 25.4 446 63.4 25 3.6 
Louisiana 14 5.0 65 23.3 170 60.9 30 10.8 
Maryland 41 6.0 211 30.8 386 56.4 46 6.7 
Massachusetts 35 6.2 179 31.5 333 58.6 21 3.7 
Mississipp1 10 11.4 23 26.1 46 52.3 9 10.2 
New Jersey 46 6.2 246 33.3 417 56.5 29 3.9 
New Mexico 2 4.3 17 37.0 26 56.5 1 2.2 
New York 3 6.3 16 33.3 27 56.3 2 4.2 
Ohio 23 10.0 51 22.3 144 62.9 11 4.8 
Rhode Island 4 3.2 39 31.2 78 62.4 4 3.2 
Tennessee 7 5.7 35 28.5 73 59.3 8 6.5 
All States 285 6.5 1274 29.0 3607 59.4 225 5.1 


Note. Numbers and percentages are based on the total number of valid responses (missing = 244). 


Table 18. TA Survey: Q1-4 Scale Means for Test Administration Experience 


State Mean Standard Deviation 
Arizona 2.78 0.51 
Arkansas 2.58 0.57 
Colorado 2.54 0.53 
D.C. 2.73 0.49 
Florida 2.82 0.67 
Illinois 2.68 0.54 
Louisiana 2.83 0.61 
Maryland 2.68 0.58 
Massachusetts 2.70 0.49 
Mississippi 2.73 0.64 
New Jersey 2.63 0.53 
New Mexico 2.63 0.44 
New York 2.71 0.48 
Ohio 2.63 0.59 
Rhode Island 2.57 0.51 
Tennessee 2.72 0.51 
All States 2.67 0.55 


Note. Mean scale scores computed by taking the mean of items 1-4, which were rated on a 4-point agreement scale 
where | = Strongly Disagree and 4 = Strongly Agree. Overall internal consistency reliability estimate is a = .82. 
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Table 19. TA Survey: Q6. Please identify the types of technology-related problems that occurred during test 


administration (select all that apply °). If a problem occurred, indicate how it was resolved. 


No technology-related problems happened during the test. 


Selected 

State n % 

Arizona 41 13.9 
Arkansas 45 14.3 
Colorado 16 10.5 
D.C. 4 12.9 
Florida 2 28.6 
Illinois 91 12.3 
Louisiana 45 14.8 
Maryland 90 12.7 
Massachusetts 74 12.3 
Mississippi 13 13.8 
New Jersey 111 14.1 
New Mexico 4 8.3 
New York 7 13.0 
Ohio 27 11.3 
Rhode Island 12 9.4 
Tennessee 7 5.5 
All States 589 12.8 


Note. Numbers and percentages are collapsed across items with variable names CBT_NoProb_Occur to CBT_NoProb_NotRes. 
“ Percentages reported for “select all that apply” items should be interpreted with caution, as it is not possible to differentiate 


“missing” (i.e., skipped) vs. “not selected” responses. 


Table 20. TA Survey: Q6. Please identify the types of technology-related problems that occurred during test 


administration (select all that apply °). If a problem occurred, indicate how it was resolved. 


Students had difficulty logging into the assessment. 


Problem Occurred Resolved Problem Resolved Problem — Problem was not 
During Administration on My Own with Assistance Resolved 
State n % n % n % n % 
Arizona (n = 254) 137 53.9 62 24.4 136 53.5 17 6.7 
Arkansas (n = 269) 123 45.7 63 23.4 128 47.6 13 4.8 
Colorado (n = 136) 83 61.0 47 34.6 75 55.1 8 5.9 
D.C. (n= 27) 15 55.6 6 22.2 15 55.6 1 3.7 
Florida (n = 5) 3 60.0 1 20.0 3 60.0 0 0.0 
Illinois (n = 646) 360 55.7 158 24.5 366 56.7 34 5.3 
Louisiana (n = 259) 112 43.2 53 20.5 120 46.3 10 3.9 
Maryland (n = 620) 340 54.8 163 26.3 319 51.5 15 2.4 
Massachusetts (n = 527) 286 54.3 126 23.9 310 58.8 14 2.7 
Mississippi (n = 81) 34 42.0 12 14.8 44 54.3 4 4.9 
New Jersey (n = 674) 323 47.9 131 19.4 343 50.9 15 2.2 
New Mexico (n = 44) 25 56.8 11 25.0 25 56.8 5 11.4 
New York (n = 47) 21 44.7 12 25.5 13 27 6 12.8 
Ohio (n = 213) 122 57.3 60 28.2 111 52.1 13 6.1 
Rhode Island (n = 116) 70 60.3 25 21.6 80 69.0 2 1.7 
Tennessee (n = 121) 69 57.0 35 28.9 63 52.1 5 4.1 
All States (n = 4046) 2127 52.6 965 23.9 2155 53.3 162 4.0 
(continued) 
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Table 20 cont. TA Survey: Q6. Please identify the types of technology-related problems that occurred during test 
administration (select all that apply“). If a problem occurred, indicate how it was resolved. 


Device(s) stopped working. 


Problem Occurred Resolved Problem Resolved Problem Problem was not 
During Administration on My Own with Assistance Resolved 

State n % n % n % n % 
Arizona (n = 254) 114 44.9 57 22.4 105 41.3 21 8.3 
Arkansas (n = 269) 99 36.8 4] 15.2 85 31.6 15 5.6 
Colorado (n = 136) 65 47.8 40 29.4 46 33.8 18 13.2 
D.C. (n = 27) 11 40.7 6 22.2 8 29.6 2 74 
Florida (n = 5) 4 80.0 1 20.0 3 60.0 0 0.0 
Illinois (n = 646) 254 39.3 108 16.7 238 36.8 55 8.5 
Louisiana (n = 259) 109 42.1 37 14.3 100 38.6 23 8.9 
Maryland (n = 620) 316 51.0 126 20.3 249 40.2 50 8.1 
Massachusetts (n = 527) 217 41.2 76 14.4 221 41.9 39 7.4 
Mississippi (n = 81) 36 44.4 14 17.3 37 45.7 8 9.9 
New Jersey (n = 674) 270 40.1 88 13.1 274 40.7 52 7.7 
New Mexico (n = 44) 23 52.3 9 20.5 18 40.9 6 13.6 
New York (n = 47) 22 46.8 6 12.8 20 42.6 3 6.4 
Ohio (n = 213) 103 48.4 39 18.3 83 39.0 26 12.2 
Rhode Island (n = 116) 63 54.3 20 17.2 63 54.3 7 6.0 
Tennessee (n = 121) 58 47.9 30 24.8 47 38.8 9 7.4 
All States (n = 4046) 1769 43.7 698 17.3 1601 39.6 335 8.3 
Device(s) worked slowly. 

Problem Occurred Resolved Problem Resolved Problem — Problem was not 

During Administration on My Own with Assistance Resolved 
State n % n % n % n % 
Arizona (n = 254) 85 33.5 41 16.1 53 20.9 26 10.2 
Arkansas (n = 269) 73 27.1 27 10.0 46 17.1 15 5.6 
Colorado (n = 136) 47 34.6 17 12.5 23 16.9 26 19.1 
D.C. (n= 27) 10 37.0 10 37.0 2 74 2 74 
Florida (n = 5) 3 60.0 1 20.0 2 40.0 1 20.0 
Illinois (n = 646) 201 31.1 83 12.8 130 20.1 76 11.8 
Louisiana (n = 259) 72 27.8 23 8.9 40 15.4 31 12.0 
Maryland (n = 620) 268 43.2 118 19.0 136 21.9 87 14.0 
Massachusetts (n = 527) 162 30.7 54 10.2 97 18.4 61 11.6 
Mississippi (n = 81) 20 24.7 7 8.6 11 13.6 8 9.9 
New Jersey (n = 674) 192 28.5 66 9.8 137 20.3 55 8.2 
New Mexico (n = 44) 20 45.5 5 11.4 8 18.2 9 20.5 
New York (n = 47) 19 40.4 4 8.5 4 8.5 10 21.3 
Ohio (n = 213) 77 36.2 23 10.8 44 20.7 38 17.8 
Rhode Island (n = 116) 4] 35.3 19 16.4 26 22.4 6 5.2 
Tennessee (n = 121) 52 43.0 17 14.0 29 24.0 15 12.4 
All States (n = 4046) 1347 33.3 515 12.7 792 19.6 467 11.5 
(continued) 
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Table 20 cont. TA Survey: Q6. Please identify the types of technology-related problems that occurred during test 


administration (select all that apply“). If a problem occurred, indicate how it was resolved. 


Lost internet connection during administration. 


Problem Occurred Resolved Problem Resolved Problem Problem was not 
During Administration on My Own with Assistance Resolved 

State n % n % n % n % 
Arizona (n = 254) 56 22.0 19 15 48 18.9 16 6.3 
Arkansas (n = 269) 38 14.1 15 5.6 37 13.8 2 0.7 
Colorado (n = 136) 34 25.0 15 11.0 25 18.4 6 44 
D.C. (n = 27) 3 11.1 2 74 3 11.1 0 0.0 
Florida (n = 5) 1 20.0 0 0.0 1 20.0 0 0.0 
Illinois (n = 646) 134 20.7 51 7.9 115 17.8 24 3.7 
Louisiana (n = 259) 58 22.4 31 12.0 36 13.9 12 4.6 
Maryland (n = 620) 138 22.3 51 8.2 107 17.3 17 2.7 
Massachusetts (n = 527) 83 15.7 36 6.8 76 14.4 14 2.7 
Mississippi (n = 81) 15 18.5 8 9.9 12 14.8 3 3.7 
New Jersey (n = 674) 104 15.4 40 5.9 121 18.0 16 2.4 
New Mexico (n = 44) 12 27.3 5 11.4 8 18.2 6 13.6 
New York (n = 47) 12 25.5 2 43 8 17.0 4 8.5 
Ohio (n = 213) 42 19.7 15 7.0 39 18.3 12 5.6 
Rhode Island (n = 116) 31 26.7 8 6.9 34 29.3 4 3.4 
Tennessee (n = 121) 29 24.0 11 9.1 26 21.5 6 5.0 
All States 792 19.6 309 7.6 698 17.3 142 3.5 


Note. Individuals who selected N/A for this item set (see Table 19) have been excluded from analyses. 
“ Percentages reported for “‘select all that apply” items should be interpreted with caution, as it is not possible to 


differentiate “missing” (1.e., skipped) vs. “not selected” responses. 
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Test Environment 


Table 21. TA Survey: Q1. Were there any problems in the physical environment during the test session that might 
have negatively affected student performance? 


Yes No 

State n % n % 

Arizona 54 19.1 229 80.9 
Arkansas 16 5.3 288 94.7 
Colorado 16 10.9 131 89.1 
D.C. 2 74 25 92.6 
Florida 0 0.0 7 100.0 
Illinois 104 14.7 604 85.3 
Louisiana 23 8.0 263 92.0 
Maryland 74 10.8 613 89.2 
Massachusetts 80 14.0 493 86.0 
Mississippi 12 13.5 77 86.5 
New Jersey 97 13.0 651 87.0 
New Mexico 5 10.9 41 89.1 
New York 9 17.3 43 82.7 
Ohio 41 17.6 192 82.4 
Rhode Island 22 17.6 103 82.4 
Tennessee 11 8.9 112 91.1 
All States 569 12.8 3875 87.2 


Note. Numbers and percentages are based on the total number of valid responses (missing = 191). 


Table 22. TA Survey: Q1a. Please indicate the nature of the problem in the physical environment.(select all that 
apply“) 


Computer Problems Lighting Problems Other 

State n % n % n % 

Arizona (n = 66) 21 31.8 3 7.6 35 53.0 
Arkansas (n = 26) 4 15.4 0 0.0 10 38.5 
Colorado (n = 21) 8 38.1 0 0.0 8 38.1 
D.C. (n= 6) 0 0.0 0 0.0 1 16.7 
Florida (n = 0) - - - - - - 

Illinois (n = 133) 42 31.6 1 0.8 69 51.9 
Louisiana (n = 41) 4 9.8 0 0.0 17 41.5 
Maryland (n = 97) 26 26.8 2 2.1 45 46.4 
Massachusetts (n = 108) 38 35.2 3 2.8 45 41.7 
Mississippi (n = 17) 2 11.8 0 0.0 10 58.8 
New Jersey (n = 134) 41 30.6 2 1.5 56 41.8 
New Mexico (n = 7) 1 14.3 0 0.0 5 71.4 
New York (n= 11) 7 63.6 1 9.1 3 27.3 
Ohio (n = 48) 20 41.7 1 2.1 28 58.3 
Rhode Island (n = 25) 12 48.0 2 8.0 11 44.0 
Tennessee (n = 16) 5 31.3 1 6.3 5 31.3 
All States 232 30.5 18 2.4 349 45.9 


Note. Individuals who selected ‘No’ for item 1 (see Table 21) have been excluded from analyses. Individuals with missing 
responses to item | (see note in Table 21) who responded to item 1a were included in analyses. 

“ Percentages reported for “select all that apply” items should be interpreted with caution, as it is not possible to differentiate 
“missing” (i.e., skipped) vs. “not selected” responses. 
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PARCC’s Timing Policy 


Table 23. TA Survey: Q1. Indicate the proportion of students who needed the Additional Time Allowed. 


None Less than 4 Approximately More than 4 
State n % n % n % n % 
Arizona 75 26.8 133 47.5 43 15.4 29 10.4 
Arkansas 129 42.6 138 45.5 25 8.3 11 3.6 
Colorado 42 29.2 73 50.7 19 13.2 10 6.9 
D.C. 7 26.9 7 26.9 9 34.6 3 11.5 
Florida 2 28.6 4 57.1 0 0.0 1 14.3 
Illinois 220 31.5 343 49.1 84 12.0 52 74 
Louisiana 91 32.2 139 49.1 23 8.1 30 10.6 
Maryland 260 38.3 312 46.0 64 9.4 42 6.2 
Massachusetts 156 27.3 264 46.2 79 13.8 72 12.6 
Mississippi 41 45.6 34 37.8 11 12.2 4 4.4 
New Jersey 261 35.3 334 45.2 72 9.7 72 9.7 
New Mexico 15 32.6 23 50.0 5 10.9 3 6.5 
New York 12 22.6 30 56.6 5 9.4 6 11.3 
Ohio 108 46.2 89 38.0 20 8.5 17 73 
Rhode Island 40 32.8 60 49.2 15 12.3 7 5.7 
Tennessee 44 36.1 59 48.4 8 6.6 11 9.0 
All States 1506 34.2 2044 46.4 482 10.9 371 8.4 


Note. Numbers and percentages are based on the total number of valid responses (missing = 232). 


Table 24, TA Survey: Qla. Please indicate how much additional time was used by the majority of these students. 


Less than 10 minutes Between 10 — 20 minutes More than 20 minutes 
State n % n % n % 
Arizona 38 18.9 112 55.7 51 25.4 
Arkansas 51 29.3 102 58.6 21 12.1 
Colorado 28 27.7 60 59.4 13 12.9 
D.C. 0 0.0 10 52.6 9 47.4 
Florida 0 0.0 4 80.0 1 20.0 
Illinois 108 22.5 273 57.0 98 20.5 
Louisiana 29 15.3 106 55.8 55 28.9 
Maryland 134 32.2 213 51.2 69 16.6 
Massachusetts 68 16.6 217 52.9 125 30.5 
Mississipp1 8 17.0 27 57.4 12 25.5 
New Jersey 121 26.0 246 52.8 99 21.2 
New Mexico 4 12.9 20 64.5 7 22.6 
New York 8 20.0 17 42.5 15 37.5 
Ohio 29 23.0 59 46.8 38 30.2 
Rhode Island 20 24.4 47 57.3 15 18.3 
Tennessee 23 29.5 45 57.7 10 12.8 
All States 670 23.4 1559 54.4 639 22.3 


Note. Numbers and percentages are based on the total number of valid responses (missing = 261). Individuals who 
selected ‘None’ for item | (see Table 23) have been excluded from analyses. Individuals with missing responses to 
item | (see note Table 23) who responded to item la were included in analyses. 
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Accessibility Features and Accommodations 


Table 25. TA Survey: Q1. Are you aware that there is an approved PARCC Accessibility Features and 
Accommodations Manual? 


Yes No 

State n % n % 

Arizona 183 66.3 93 33.7 
Arkansas 218 72.2 84 27.8 
Colorado 99 68.8 45 31.3 
D.C. 19 70.4 8 29.6 
Florida 3 42.9 4 57.1 
Illinois 487 69.4 215 30.6 
Louisiana 184 64.3 102 35.7 
Maryland 486 71.8 191 28.2 
Massachusetts 385 67.9 182 32.1 
Mississipp1 57 64.8 31 35.2 
New Jersey 494 67.2 241 32.8 
New Mexico 31 67.4 15 32.6 
New York 31 59.6 21 40.4 
Ohio 159 67.9 75 32.1 
Rhode Island 106 86.2 17 13.8 
Tennessee 91 74.6 31 25.4 
All States 3037 69.1 1357 30.9 


Note. Numbers and percentages are based on the total number of valid responses (missing = 241). 


Table 26. TA Survey: Qla. Did you read the “Accessibility Features and Accommodations Manual” prior to 
administration? 


Yes No 

State n % n % 

Arizona 124 67.0 61 33.0 
Arkansas 142 64.8 77 35.2 
Colorado 71 71.7 28 28.3 
D.C. 13 68.4 6 31.6 
Florida 3 100.0 0 0.0 
Illinois 328 68.2 153 31.8 
Louisiana 146 80.2 36 19.8 
Maryland 337 69.2 150 30.8 
Massachusetts 268 69.8 116 30.2 
Mississipp1 36 62.1 22 37.9 
New Jersey 347 70.2 147 29.8 
New Mexico 20 64.5 11 35.5 
New York 24 75.0 8 25.0 
Ohio 115 73.2 42 26.8 
Rhode Island 80 76.9 24 23.1 
Tennessee 60 65.9 31 34.1 
All States 2117 69.9 913 30.1 


Note. Numbers and percentages are based on the total number of valid responses (missing = 248). Individuals who 
selected ‘No’ for item 1 (see Table 25) have been excluded from analyses. Individuals with missing responses to 
item | (see note in Table 25) who responded to item 1a were included in analyses. 
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Table 27. TA Survey: Q2. Are you aware there is approved PARCC guidance on “Computer-Based Accessibility 
Features and Embedded Accommodations” ? 


Yes No 

State n % n % 

Arizona 153 55.2 124 44.8 
Arkansas 187 62.1 114 37.9 
Colorado 91 63.2 53 36.8 
D.C. 14 51.9 13 48.1 
Florida 3 42.9 4 57.1 
Illinois 391 56.3 304 43.7 
Louisiana 179 63.5 103 36.5 
Maryland 427 63.4 247 36.6 
Massachusetts 325 58.2 233 41.8 
Mississipp1 44 50.0 44 50.0 
New Jersey 382 52.4 347 47.6 
New Mexico 21 45.7 25 54.3 
New York 26 51.0 25 49.0 
Ohio 128 55.4 103 44.6 
Rhode Island 88 73.3 32 26.7 
Tennessee 77 62.6 46 37.4 
All States 2540 58.3 1819 41.7 


Note. Numbers and percentages are based on the total number of valid responses (missing = 276). 


Table 28. TA Survey: Q2a. Did you read the “Computer-Based Accessibility Features and Embedded 
Accommodations Available for PARCC Field Test” ? 


Yes No 

State n % n % 

Arizona 102 68.5 47 31.5 
Arkansas 129 69.7 56 30.3 
Colorado 69 77.5 20 22.5 
D.C. 12 92.3 1 TA 
Florida 3 100.0 0 0.0 
Illinois 271 68.8 123 31.2 
Louisiana 148 81.8 33 18.2 
Maryland 321 76.2 100 23.8 
Massachusetts 230 69.9 99 30.1 
Mississippi 29 64.4 16 35.6 
New Jersey 256 68.1 120 31.9 
New Mexico 15 71.4 6 28.6 
New York 20 76.9 6 23.1 
Ohio 91 74.0 32 26.0 
Rhode Island 70 77.8 20 22.2 
Tennessee 52 69.3 23 30.7 
All States 1821 72.1 703 27.9 


Note. Numbers and percentages are based on the total number of valid responses (missing = 292). Individuals who 
selected ‘No’ for item 2 (see Table 27) have been excluded from analyses. Individuals with missing responses to 
item 2 (see note in Table 27) that responded to item 2a were included in analyses. 
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Table 29. TA Survey: Q3. Indicate whether you administered any of the following 
accessibility features or accommodations to your students during the Field Test 
Administration (select all that apply“). 


Not Applicable 

State n % 

Arizona 135 45.8 
Arkansas 163 51.9 
Colorado 82 53.9 
D.C. 11 35.5 
Florida 2 28.6 
Illinois 367 49.8 
Louisiana 155 51.0 
Maryland 323 45.5 
Massachusetts 298 49.6 
Mississipp1 48 51.1 
New Jersey 423 53.9 
New Mexico 30 62.5 
New York ai 50.0 
Ohio 121 50.4 
Rhode Island 77 60.2 
Tennessee 71 55.5 
All States 2335 50.4 


Note. * Percentages reported for “select all that apply” items should be interpreted with caution, as it is not possible 
to differentiate “missing” (i.e., skipped) vs. “not selected” responses. 
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Table 30. TA Survey: Q3. Indicate whether you administered any of the following accessibility features or accommodations to your students during the Field 


Test Administration (select all that apply“). 


Clarify Repeat Redirect a Human Reader Human Reader 
Directions Directions Student Math English Scribe English Scribe Math 

State n % n % n % n n % n % n % n 

Arizona (n= 160) 82 51.2 76 47.5 78 48.8 18 11.3 2 1.3 2 1.3 1 0.6 
Arkansas (n = 151) 53 35.1 72 47.7 52 34.4 17 11.3 12 7.9 0 0.0 0 0.0 
Colorado (n = 70) 37 52.9 36 51.4 35 50.0 4 5:7 3 4.3 2 2.9 1 1.4 
D.C. (n = 20) 8 40.0 12 60.0 9 45.0 4 20.0 0 0.0 0 0.0 0 0.0 
Florida (n = 5) 2 40.0 1 20.0 0 0.0 0 0.0 1 20.0 1 20.0 0 0.0 
Illinois (n = 370) 192 51.9 177 47.8 173 46.8 46 12.4 19 5.1 16 4.3 12 3.2 
Louisiana (n = 149) 55 36.9 87 58.4 57 38.3 23 15.4 29 19.5 5 3.4 2 1.3 
Maryland (n = 387) 109 28.2 162 41.9 134 34.6 75 19.4 96 24.8 72 18.6 34 8.8 
Massachusetts (n= 303) 178 58.7 137 45.2 140 46.2 42 13.9 13 4.3 14 4.6 10 3.3 
Mississippi (n = 46) 9 19.6 14 30.4 19 41.3 8 17.4 6 13.0 0 0.0 0 0.0 
New Jersey (n = 362) 194 53.6 191 52.8 138 38.1 31 8.6 33 9.1 9 25 3 0.8 
New Mexico (n = 18) 5 27.8 9 50.0 6 33.3 2 11.1 1 5.6 1 5.6 0 0.0 
New York (n = 27) 14 51.9 16 59.3 14 51.9 7 25.9 2 7.4 0 0.0 1 3.7 
Ohio (n = 119) 60 50.4 57 47.9 47 39.5 17 14.3 15 12.6 3 2.5 5 4.2 
Rhode Island (n = 51) 30 58.8 22 43.1 20 39.2 a 13.7 4 7.8 2 3.9 3 5.9 
Tennessee (n = 57) 20 35.1 14 24.6 14 24.6 7 12.3 2) 8.8 0 0.0 0 0.0 
All States (n = 2300) 1049 45.6 1086 47.2 940 40.9 308 13.4 242 10.5 127 5.5 72 3.1 


Note. Individuals who selected N/A for this item (see Table 29) have been excluded from analyses. 


“ Percentages reported for “‘select all that apply” items should be interpreted with caution, as it is not possible to differentiate “missing” (i.e., skipped) vs. “not 


selected” responses. 
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Table 31. TA Survey: Q4. If you served as a human reader (read text aloud) to a student 
and/or served as a scribe, please indicate if you read the following documents (select all 


that apply“): 


Not Applicable 

State n % 

Arizona 199 67.5 
Arkansas 232 73.9 
Colorado 115 154 
D.C. 15 48.4 
Florida 4 57.1 
Illinois 522 70.8 
Louisiana 208 68.4 
Maryland 456 64.2 
Massachusetts 414 68.9 
Mississipp1 65 69.1 
New Jersey 542 69.0 
New Mexico 40 83.3 
New York 38 70.4 
Ohio 164 68.3 
Rhode Island 94 73.4 
Tennessee 100 78.1 
All States 3212 69.3 


Note. * Percentages reported for “select all that apply” items should be interpreted with caution, as it is not possible 
to differentiate “missing” (i.e., skipped) vs. “not selected” responses. 


Table 32. TA Survey: Q4. If you served as a human reader (read text aloud) to a student and/or served as a scribe, 
please indicate if you read the following documents (select all that apply“): 


Appendix B Appendix C Appendix I Appendix J 
State n % n % n % n % 
Arizona (n = 160) 16 16.7 0 0.0 0 0.0 9 9.4 
Arkansas (n = 151) 20 24.4 1 1.2 q 8.5 6 7.3 
Colorado (n = 70) 3 8.1 2 5.4 1 2.7 1 27 
D.C. (n = 20) 1 6.3 0 0.0 1 6.3 1 6.3 
Florida (n = 5) 0 0.0 0 0.0 0 0.0 0 0.0 
Illinois (n = 370) 28 13.0 13 6.0 6 2.8 18 8.4 
Louisiana (n = 149) 29 30.2 8 8.3 13 13.5 8 8.3 
Maryland (n = 387) 72 28.3 65 25.6 57 22.4 35 13.8 
Massachusetts (n = 303) 34 18.2 14 75 11 5.9 24 12.8 
Mississippi (n = 46) 4 13.8 0 0.0 1 3.4 4 13.8 
New Jersey (n = 362) 41 16.9 7 2.9 10 4.1 9 3.7 
New Mexico (n = 18) 1 12.5 1 12.5 1 12.5 1 12.5 
New York (n = 27) 3 18.8 0 0.0 0 0.0 4 25.0 
Ohio (n= 119) 19 25.0 12 15.8 7 9.2 10 13.2 
Rhode Island (n = 51) 6 17.6 0 0.0 3 8.8 2 5.9 
Tennessee (n = 57) 5 17.9 3 10.7 3 10.7 3 10.7 
All States (n = 1423) 283 19.9 126 8.9 122 8.6 135 9.5 


Note. Individuals who selected N/A for this item (see Table 31) have been excluded from analyses. 
“ Percentages reported for “‘select all that apply” items should be interpreted with caution, as it is not possible to 


differentiate “missing” (i.e., skipped) vs. “not selected” responses. 
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Table 33. TA Survey: Q5. Did you receive training on administering accessibility features/accommodations from 
your school or district? 


Yes No 

State n % n % 

Arizona 88 34.0 171 66.0 
Arkansas 158 56.2 123 43.8 
Colorado 52 39.1 81 60.9 
D.C. 12 48.0 13 52.0 
Florida 2 50.0 2 50.0 
Illinois 175 26.3 491 73.7 
Louisiana 155 55.8 123 44.2 
Maryland 384 59.2 265 40.8 
Massachusetts 180 33.1 363 66.9 
Mississipp1 34 40.0 51 60.0 
New Jersey 276 39.2 428 60.8 
New Mexico 10 22.2 35 77.8 
New York 11 22.0 39 78.0 
Ohio 70 32.9 143 67.1 
Rhode Island 36 30.5 82 69.5 
Tennessee 35 30.7 79 69.3 
All States 1681 40.3 2492 59.7 


Note. Numbers and percentages are based on the total number of valid responses (missing = 462). 
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Sample Items/Tutorials 


Table 34. TA Survey: Q1. Did the students in your session(s) practice with PARCC sample items prior to 
administration? 


Yes No Not Sure/Don’t Know 
State n % n % n % 
Arizona 206 78.0 40 15.2 18 6.8 
Arkansas 242 83.7 28 9.7 19 6.6 
Colorado 102 74.5 25 18.2 10 7.3 
D.C. 18 72.0 4 16.0 3 12.0 
Florida 4 100 0 0.0 0 0.0 
Illinois 534 78.8 88 13.0 56 8.3 
Louisiana 203 72.2 41 14.6 37 13.2 
Maryland 500 76.2 84 12.8 72 11.0 
Massachusetts 430 78.8 a7 10.4 59 10.8 
Mississippi 54 62.1 16 18.4 17 19.5 
New Jersey 589 81.4 64 8.8 71 9.8 
New Mexico 35 77.8 8 17.8 2 4.4 
New York 36 73.5 10 20.4 3 6.1 
Ohio 150 67.6 52 23.4 20 9.0 
Rhode Island 102 85.7 12 10.1 5 4.2 
Tennessee 71 62.3 31 DTZ 12 10.5 
All States 3281 77.3 561 13.2 404 9.5 


Note. Numbers and percentages are based on the total number of valid responses (missing = 389). 


Table 35. TA Survey: Q2. Did the students in your session(s) complete the PARCC tutorial prior to administration? 


Yes No Not Sure/Don’t Know 
State n % n % n % 
Arizona 208 78.8 34 12.9 22 8.3 
Arkansas 259 88.7 15 5.1 18 6.2 
Colorado 61 43.9 42 30.2 36 25.9 
D.C. 13 52.0 7 28.0 5 20.0 
Florida 3 75.0 0 0.0 1 25.0 
Illinois 485 71.5 109 16.1 84 12.4 
Louisiana 214 75.9 36 12.8 32 11.3 
Maryland 457 69.6 86 13.1 114 17.4 
Massachusetts 419 77.2 59 10.9 65 12.0 
Mississippi 59 68.6 11 12.8 16 18.6 
New Jersey 593 82.0 53 7.3 77 10.7 
New Mexico 30 66.7 14 31.1 1 2.2 
New York 35 70.0 8 16.0 7 14.0 
Ohio 150 68.2 47 21.4 23 10.5 
Rhode Island 96 80.7 13 10.9 10 8.4 
Tennessee 61 53.0 38 33.0 16 13.9 
All States 3146 74.1 574 13.5 528 12.4 


Note. Numbers and percentages are based on the total number of valid responses (missing = 387). 
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Table 36. TA Survey: Q3. In accordance with your state or local policy, please indicate which of the following 
options was given to students after the completion of testing. 


Quiet Recreational 


Sit Quietly Dismissed Material 

State n % n % n % 

Arizona 80 30.4 30 11.4 153 58.2 
Arkansas 166 56.8 28 9.6 98 33.6 
Colorado 58 41.7 7 5.0 74 53.2 
D.C. 6 25.0 2 8.3 16 66.7 
Florida 3 75.0 0 0.0 1 25.0 
Illinois 157 23.2 34 5.0 485 71.7 
Louisiana 97 34.9 21 7.6 160 57.6 
Maryland 383 58.4 85 13.0 188 28.7 
Massachusetts 128 23.4 82 15.0 336 61.5 
Mississipp1 57 66.3 4 4.7 25 29.1 
New Jersey 336 46.7 76 10.6 308 42.8 
New Mexico 16 35.6 3 6.7 26 57.8 
New York 12 23.5 10 19.6 29 56.9 
Ohio 74 34.1 35 16.1 108 49.8 
Rhode Island 32 27.1 8 6.8 78 66.1 
Tennessee 54 47.0 10 8.7 51 44.3 
All States 1661 39.2 436 10.3 2138 50.5 


Note. Numbers and percentages are based on the total number of valid responses (missing = 400). 
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Process for Cleaning Up Data for PARCC TA Survey 


e Initial dataset had 8736 respondents, which included responses from Test 
Administrators who administered the computer/tablet-based test (CBT) and Test 
Administrators who administered the paper-based test (PBT). 


Clean-up Steps 
1) Sorted data by start date and deleted cases that began before the initial launch of the 


survey (3/24/14), as well as cases that included words such as ‘test’, ‘delete’, or 
‘dummy’ in the text box items. 
O n=8729 
= CBT respondents = 6478 
= PBT respondents = 2251 


2) Checked for duplicate cases within the dataset; none were found. 
O n=8729 


3) Individuals who only responded to the first five background items (1-4b) and no other 
substantive survey content were removed. 
O n=8583 
=" CBT respondents = 6362 
= PBT respondents = 2221 


4) Further examined survey content items for each assessment type (CBT and PBT) to 
identify participants who failed to respond to any substantive content on the survey. So 
as not to exclude any meaningful responses, a conservative guideline was applied for 
removing cases; only individuals with 2 or fewer responses (not including background 
items) were removed. 

O n=6322 
= CBT respondents = 4635 
= PBT respondents = 1687 


5) Checked for pattern responding. Computed the number of times individuals used a 
particular response option (i.e., response 1, response 2, etc.) to help determine whether 
respondent provided thoughtful responses or was simply selecting the first response 
option for every item on the survey, for example. No evidence of pattern responding 
was found. Consequently, no cases were removed at this step. 

O n=6322 
= CBT respondents = 4635 
= PBT respondents = 1687 
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Appendix J — Item-level Results from Test Administrator Survey: 
Paper-based Administration 


Administration Information by State 


Table 1. TA Survey: Q1. Please indicate the mode of assessment administration: 


Computer-based (or tablet) Administration _Paper-and-pencil Administration 


State n % n % 

Arizona (N = 493) 295 59.8 198 40.2 
Arkansas (N = 471) 314 67.0 157 33.0 
Colorado (N = 223) 152 68.2 71 31.8 
District of Columbia (D.C.; N = 35) 31 89.0 4 11.0 
Florida (N = 8) 7 87.5 1 12.5 
Illinois (N = 1024) 737 72.0 287 28.0 
Louisiana (N = 311) 304 98.0 7 2.0 
Maryland (N = 1171) 710 60.6 461 39.4 
Massachusetts (N = 776) 601 774 175 22.6 
Mississippi (N = 148) 94 63.5 54 36.5 
New Jersey (N = 788) 785 99.6 3 0.4 
New Mexico (N = 48) 48 100.0 0 0.0 
New York (N = 57) 54 94.7 3 5.3 
Ohio (N = 393) 240 61.1 153 38.9 
Rhode Island (N = 183) 128 69.9 55 30.1 
Tennessee (N = 183) 128 69.9 55 30.1 
All States (N = 6322) 4635° 73.3 1687° 26.7 


Note. Percentages are based on the total number of valid responses for each state. Number of missing for computer- 
based administration = 7. Number of missing for paper-and-pencil administration = 3. 

“Includes the 7 cases that did not indicate their state. 

° Includes the 3 cases that did not indicate their state. 
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Table 2. TA Survey: Q4-5. Please identify the grade level(s)/course(s) in which you administered the assessment(s) 
(select all that apply“). 


Not Applicable 
ELA/L Performance-based ELA/L End-of-Year Math Performance- Math End-of-Year 
Assessment Assessment based Assessment Assessment 

State n % n % n % n % 
Arizona 35 17.7 33 16.7 31 15.7 26 13.1 
Arkansas 45 28.7 32 20.4 36 22.9 30 19.1 
Colorado 9 12.7 6 8.5 17 23.9 11 15.5 
D.C. 4 100.0 2 50.0 3 75.0 2 50.0 
Florida 0 0.0 0 0.0 1 100.0 0 0.0 
Illinois 69 24.0 64 22.3 66 23.0 48 16.7 
Louisiana 2 28.6 2 28.6 1 14.3 0 0.0 
Maryland 125 27.1 90 19.5 99 21.5 75 16.3 
Massachusetts 57 32.6 55 31.4 25 14.3 20 11.4 
Mississippi 20 37.0 14 25.9 6 11.1 4 74 
New Jersey 1 33.3 0 0.0 2 66.7 1 33.3 
New York 0 0.0 0 0.0 1 33.3 0 0.0 
Ohio 45 29.4 37 24.2 19 12.4 24 15.7 
Rhode Island 14 25.5 14 25.5 8 14.5 8 14.5 
Tennessee 16 29.1 10 18.2 11 20.0 8 14.5 
All States 442 26.2 359 21.3 327 19.4 257 15.2 


Note. * Percentages reported for “select all that apply” items should be interpreted with caution, as it is not possible 
to differentiate “missing” (i.e., skipped) vs. “not selected” responses. 


Updated February 17, 2015 Page J-2 


PARCC 


Quality of Test Administration 


Table 3. TA Survey: Q4. Please identify the grade level(s)/course(s) in which you administered the assessment(s) (select all that apply “). 


ELA/L Performance-Based Assessment 


Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10 Grade 11 
State n % n % n % n % n % n % n % n % n % 
Arizona (n = 163) 9 5.5 8 4.9 13 8.0 9 5.5 11 6.7 8 4.9 3 1.8 12 7.4 1 0.6 
Arkansas (n = 112) 10 8.9 9 8.0 5 4.5 1 0.9 7 6.3 8 7.1 0 0.0 2 1.8 4 3.6 
Colorado (n = 62) 14 22.6 10 16.1 8 12.9 2 3.2 0 0.0 2 3.2 3 4.8 3 4.8 3 4.8 
D.C. (n= 0) -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
Florida (n = 1) 1 100.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0.0 
Illinois (n = 218) 14 6.4 26 11.9 20 9.2 15 6.9 21 9.6 19 8.7 8 3.7 8 3.7 3.7 
Louisiana (n = 5) 0 0.0 0 0.0 0 0.0 1 = 20.0 0 0.0 0 0.0 0 0.0 0 0.0 0.0 
Maryland (n = 336) 48 143 53 15.8 0 0.0 15 4.5 34 10.1 0 0.0 26 7.7 6 18 43 12.8 
Massachusetts (n=118) 7 5.9 5 4.2 6 5.1 3 2.5 13 11.0 3 2.5 3 2:5 2 1.7 2 1.7 
Mississippi (n = 34) 0 0.0 7 20.6 1 2.9 5 14.7 1 2.9 1 2.9 2 5.9 6 17.6 1 2.9 
New Jersey (n = 2) 0 0.0 0 0.0 0 0.0 0 0.0 1 50.0 0 0.0 0 0.0 0 0.0 0 0.0 
New York (n= 3) 0 0.0 0 0.0 1 33.3 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 
Ohio (n = 108) 8 7.4 0 0.0 2 1.9 5 4.6 5 4.6 4 3.7 12 11.1 11 10.2 12 11.1 
Rhode Island (n = 41) 2 4.9 4 9.8 8 19.5 1 2.4 3 7.3 1 2.4 2 4.9 0 0.0 0 0.0 
Tennessee (n = 39) 3 7.7 1 2.6 4 10.3 2 5.1 6 15.4 5 12.8 2 5.1 0 0.0 0 0.0 
All States (n = 1245) 117 94 123 9.9 68 5.5 60 48 102 8.2 51 4.1 61 4.9 50 40 74 5.9 


Note. Individuals who selected N/A for this item (see Table 2) have been excluded from analyses. 
“ Percentages reported for “select all that apply” items should be interpreted with caution, as it is not possible to differentiate “missing” (i.e., skipped) vs. “not 


selected” responses. 
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Table 4. TA Survey: Q4. Please identify the grade level(s)/course(s) in which you administered the assessment(s) (select all that apply “). 


ELA/L End-of-Year Assessment 


Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10 Grade 11 
State n % n % n % n % n % n % n % n % n % 
Arizona (n= 165) 10 6.1 10 6.1 5 3.0 8 4.8 5 3.0 1 0.6 1 0.6 1 0.6 1 0.6 
Arkansas (n = 125) 7 5.6 9 7.2 6 4.8 2 1.6 2 1.3 7 5.6 1 0.8 3 2.4 9 7.2 
Colorado (n = 65) 4 6.2 0 0.0 1 1.5 0 0.0 2 3.1 1 1.5 3 4.6 0 0.0 0 0.0 
D.C. (n= 2) 0 0.0 2 100.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 
Florida (n= 1) 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 
Illinois (n = 223) 7 3.1 8 3.6 10 4.5 3 1.3 2 0.9 4 1.8 3 1.3 6 2 3 1.3 
Louisiana (n = 5) 0 0.0 2 40.0 0 0.0 1 20.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 
Maryland (n = 371) 15 4.0 15 4.0 1 0.3 1 0.3 13 3.5 1 0.3 5 1.3 2 0.5 9 2.4 
Massachusetts (n=120) 3 2.5 4 3.3 2 1.7 4 3.3 2 1.7 3 2.5 0 0.0 1 0.8 0 0.0 
Mississippi (n = 40) 0 0.0 0 0.0 0 0.0 2 5.0 0 0.0 2 5.0 1 2.5 3 75 0 0.0 
New Jersey (n = 3) 0 0.0 1 33.3 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 
New York (n = 3) 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 
Ohio (n = 116) 5 43 2 1.7 2 1.7 1 0.9 1 0.9 1 0.9 2 1.7 2 1.7 3 2.6 
Rhode Island (n = 41) 2 4.9 0 0.0 4 9.8 3 7.3 2 4.9 0 0.0 2 4.9 0 0.0 0 0.0 
Tennessee (n = 45) 4 8.9 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 1 2.2 
All States (n = 1328) 57 43 53 4.0 31 2.3 25 1.9 29 2.2 20 1.5 18 1.4 18 1.4 26 2.0 


Note. Individuals who selected N/A for this item (see Table 2) have been excluded from analyses. 
“ Percentages reported for “‘select all that apply” items should be interpreted with caution, as it is not possible to differentiate “missing” (i.e., skipped) vs. “not 


selected” responses. 
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Table 5. TA Survey: Q5. Please identify the grade level(s)/course(s) in which you administered the assessment(s) (select all that apply “). 
Math Performance-Based Assessment 
Grade 3 Grade4  Grade5  Grade6 Grade7 Grade8 ALGI GEO ALG 2 INT 1 INT 2 INT 3 


State n %* n % n % nn % nn % nn %® nn %M nn %® nH % n % Dn % TH GU 

Arizona (n = 167) ll 66 19 114 18 108 5 30 12 72 4 24 6 36 1 06 0 00 0 00 0 00 0 0.0 
Arkansas (n = 121) 10 83 14 116 11 91 4 #33 #6 50 6 50 3 25 2 #17 #2 #17 40 00 0 00 0 0.0 
Colorado (n = 54) 7 130 5 93 2 37 1 #19 #3 56 4 74 2 37 0 00 0 00 0 00 0 00 0 0.0 
D.C. (n= 1) 1 1000 0 00 0 00 0 00 0 00 0 00 0 00 0 00 0 00 0 00 0 00 0 - 0.0 
Florida (n = 0) we “oe wo woe wo -- wo -- --oe we 

Illinois (n = 221) 19 86 11 50 20 90 14 63 16 72 14 63 11 50 10 45 11 50 2 09 O 00 0O- 0.0 
Louisiana (n = 6) 0 00 0 00 0 00 1 167 0 00 0 00 1 167 1 167 #1 #167 0 #00 0 00 O- 0.0 
Maryland (n = 362) 44 12.2 82 22.7 0 0.0 47 130 44 122 0 00 1 03 11 30 10 28 0 00 0 00 0 - 0.0 
Massachusetts (n=150) 9 60 11 73 6 40 17 113 11 73 6 40 2 13 2 13 0 00 0 00 0 00 3 = 2.0 
Mississippi (n = 48) 6 125 5 104 2 42 4 83 1 21 0 00 0 00 0 00 3 63 0 00 0 00 0 0.0 
New Jersey (n = 1) 0 00 0 00 0 00 0 00 0 00 0 00 0 00 0 00 0 00 0 00 0 00 0 0.0 
New York (n = 2) 0 00 0 00 0 00 0 00 1 500 0 00 0 00 0 00 0 00 0 00 0 00 0. 0.0 
Ohio (n = 134) 5 37 7 52 6 45 7 52 18 134 13 97 6 45 5 3.7 7 52 9 67 6 45 #2 «14 
Rhode Island (n = 47) 1 21 #5 106 5 106 4 85 1 21 2 43 1 21 0 00 0 00 0 00 0 00 0 - 0.0 
Tennessee (n = 44) 3 68 %7 159 1 23 6 136 4 91 4 91 0 00 2 45 0 00 0 00 0 00 0 0.0 
All States (n= 1360) 117 86 166 12.2 71 5.2 110 81 117 86 53 39 33 24 34 25 34 25 11 08 6 04 5 04 


Note. Individuals who selected N/A for this item (see Table 2) have been excluded from analyses. 
“ Percentages reported for “select all that apply” items should be interpreted with caution, as it is not possible to differentiate “missing” (i.e., skipped) vs. “not 
selected” responses. 
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Table 6. TA Survey: Q5. Please identify the grade level(s)/course(s) in which you administered the assessment(s) (select all that apply“). 


Math End-of-Year Assessment 


Grade 3 Grade4GradeS  Grade6 Grade7 Grade8 ALGI1 GEO ALG 2 INT1 INT2 — INT3 
State n % Nn % n % on % n % n % n % nn % GU % n % nn %N 
Arizona (n= 172) 11 64 23 134 12 70 11 64 #9 52 2 12 3 #17 0 00 1 #06 0 00 0 00 0 0.0 
Arkansas (n = 127) 5 39 4 31 #4 31 4 31 12 94 9 71 2 16 3 24 0 00 0 00 0 00 O 0.0 
Colorado (n = 60) 3 50 4 67 0 00 2 33 0 00 1 17 0 00 0 00 0 00 0 00 0 00 0 0.0 
D.C. (n = 2) 1 500 0 00 0 00 0 0.0 0 00 0 00 0 00 0 00 0 00 0 00 0 00 0 0.0 
Florida (n = 1) 0 00 0 00 0 00 0 00 0 00 0 00 0 00 0 00 0 00 0 00 0 00 0 0.0 
Illinois (n = 239) 10 42 4 17 #14 59 5 21 4 17 3 13 11 46 12 50 13 54 0 00 0 00 0 0.0 
Louisiana (n = 7) 0 00 4 571 #0 00 2 286 0 00 1 143 0 00 0 00 0 00 0 00 0 00 0 0.0 
Maryland (n = 371) 15 39 31 80 0 00 10 26 11 28 0 00 0 00 9 23 2 05 0 00 0 00 0 0.0 
Massachusetts (n=155) 14 90 9 58 10 65 18 116 11 71 #5 32 1 06 4 26 4 26 1 06 0 00 1 0.6 
Mississippi (n = 50) 2 40 1 20 #4 80 2 40 3 60 1 20 2 40 0 00 3 60 0 00 0 00 0 0.0 
New Jersey (n = 2) 0 00 0 00 0 00 0 00 0 00 0 00 0 00 0 00 0 00 0 00 0 00 0 0.0 
New York (n= 3) 0 00 0 O00 1 333 0 00 0 00 0 00 0 00 0 00 0 00 0 00 0 00 0 0.0 
Ohio (n = 129) 7 54 5 39 1 O08 4 3.1 7 54 4 31 1 O08 4 31 3 23 0 00 0 00 2 1.6 
Rhode Island (n = 44) 3 64 0 00 5 106 2 43 4 85 1 21 1 21 5 106 2 43 0 00 0 00 0 0.0 
Tennessee (n = 47) 2 43 4 85 0 00 2 43 0 00 1 21 0 00 0 00 0 00 0 00 0 00 0 0.0 
All States (n = 1430) 73 5.1 89 62 S51 36 62 43 61 43 28 #%20 21 #415 #37 #26 28 20 1 01 0 00 3 02 


Note. Individuals who selected N/A for this item (see Table 2) have been excluded from analyses. 
“ Percentages reported for “‘select all that apply” items should be interpreted with caution, as it is not possible to differentiate “missing” (i.e., skipped) vs. “not 


selected” responses. 
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Test Administrator’s Training Experience 


Table 7. TA Survey: Q1. Did you receive any training on administering the assessments 
from your state, school or district?(select all that apply“): 


No, I did not receive training from my state, district or school 


State n % 

Arizona 27 13.6 
Arkansas 4 2:5 
Colorado 6 8.5 
D.C. 0 0.0 
Florida 0 0.0 
Illinois 80 27.9 
Louisiana 1 14.3 
Maryland 12 2.6 
Massachusetts 28 16.0 
Mississipp1 1 1.9 
New Jersey 0 0.0 
New York 1 33.3 
Ohio 33 21.6 
Rhode Island 12 21.8 
Tennessee 10 18.2 
All States 215 12.7 


Note. * Percentages reported for “select all that apply” items should be interpreted with caution, as it is not possible 
to differentiate “missing” (i.e., skipped) vs. “not selected” responses. 


Table 8. TA Survey: Q1. Did you receive training from your state, school or district ?(select all that apply“): 


Training from State Training from District Training from School 
State n % n % n % 
Arizona (n= 171) 2 1.2 64 37.4 113 66.1 
Arkansas (n = 153) 5 3.3 48 31.4 121 79.1 
Colorado (n = 65) 4 6.2 24 36.9 41 63.1 
D.C. (n= 4) 0 0.0 0 0.0 3 75.0 
Florida (n = 1) 0 0.0 0 0.0 1 100.0 
Illinois (n = 207) 8 3.9 105 50.7 124 59.9 
Louisiana (n = 6) 1 16.7 1 16.7 3 50.0 
Maryland (n = 449) 9 2.0 142 31.6 337 75.1 
Massachusetts (n = 147) 8 5.4 25 17.0 121 82.3 
Mississippi (n = 53) 1 1.9 10 18.9 51 96.2 
New Jersey (n = 3) 0 0.0 1 33.3 2 66.7 
New York (n= 2) 0 0.0 0 0.0 1 50.0 
Ohio (n = 120) 6 5.0 38 31.7 93 TLS 
Rhode Island (n = 43) 10 23.3 14 32.6 29 67.4 
Tennessee (n = 45) 2 4.4 10 22.2 42 93.3 
All States 56 3.8 483 32.8 1084 73.6 


Note. Individuals who selected ‘None’ for this item (see Table 7) have been excluded from analyses. 
“ Percentages reported for “‘select all that apply” items should be interpreted with caution, as it is not possible to 
differentiate “missing” (i.e., skipped) vs. “not selected” responses. 
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Test Administration Experience 


Table 9. TA Survey: Q1. The policies and procedures within the Test Administrator Manual were easy for me to 
understand. 


Strongly Disagree Disagree Agree Strongly Agree 
State n % n % n % n % 
Arizona 14 7.1 67 34.2 108 55.1 7 3.6 
Arkansas 14 9.0 36 23.2 96 61.9 9 5.8 
Colorado 5 7.1 24 34.3 39 55.7 2 2.9 
D.C. 0 0.0 0 0.0 3 75.0 1 25.0 
Florida 0 0.0 0 0.0 1 100.0 0 0.0 
Illinois 12 4.2 75 26.1 182 63.4 18 6.3 
Louisiana 1 14.3 2 28.6 3 42.9 1 14.3 
Maryland 30 6.6 113 24.9 260 57.4 50 11.0 
Massachusetts 7 4.0 51 29.1 105 60.0 12 6.9 
Mississipp1 5 9.4 19 35.8 23 43.4 6 11.3 
New Jersey 2 66.7 0 0.0 1 33.3 0 0.0 
New York 0 0.0 0 0.0 2 66.7 1 33.3 
Ohio 7 4.6 31 20.4 102 67.1 12 7.9 
Rhode Island 2 3.7 13 24.1 34 63.0 5 9.3 
Tennessee 2 3.6 16 29.1 31 56.4 6 10.9 
All States 101 6.0 447 26.8 993 59.4 130 7.8 


Note. Numbers and percentages are based on the total number of valid responses (missing = 16). 


Table 10. TA Survey: Q2. The instructions (including scripts for administering the assessment) within the Test 
Administrator Manual were easy for me to implement. 


Strongly Disagree Disagree Agree Strongly Agree 
State n % n % n % n % 
Arizona 26 13.4 82 42.3 75 38.7 11 5.7 
Arkansas 30 19.5 53 34.4 65 42.2 6 3.9 
Colorado 12 16.9 19 26.8 38 53.5 2 2.8 
D.C. 0 0.0 0 0.0 3 75.0 1 25.0 
Florida 0 0.0 0 0.0 1 100.0 0 0.0 
Illinois 19 6.6 113 39.5 136 47.6 18 6.3 
Louisiana 2 28.6 2 28.6 3 42.9 0 0.0 
Maryland 39 8.6 146 32.2 226 49.8 43 9.5 
Massachusetts 10 5.7 58 33.3 96 55.2 10 Bee 
Mississippi 7 13.0 18 33.3 23 42.6 6 11.1 
New Jersey 2 66.7 0 0.0 1 33.3 0 0.0 
New York 1 33.3 0 0.0 1 33.3 1 33.3 
Ohio 12 7.9 56 36.8 74 48.7 10 6.6 
Rhode Island 5 9.4 16 30.2 29 54.7 3 5.7 
Tennessee 3 535 16 29.1 32 58.2 4 7.3 
All States 169 10.1 579 34.7 805 48.3 115 6.9 


Note. Numbers and percentages are based on the total number of valid responses (missing = 19). 
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Table 11. TA Survey: Q3. Students appeared to understand the instructions I read to them during test 
administration. 


Strongly Disagree Disagree Agree Strongly Agree 
State n % n % n % n % 
Arizona 47 24.4 89 46.1 50 25.9 7 3.6 
Arkansas 29 19.0 54 35.3 61 39.9 9 5.9 
Colorado 18 25.4 27 38.0 23 32.4 3 4.2 
D.C. 0 0.0 1 25.0 3 75.0 0 0.0 
Florida 0 0.0 0 0.0 1 100.0 0 0.0 
Illinois 38 13.4 88 31.0 138 48.6 20 7.0 
Louisiana 1 14.3 3 42.9 3 42.9 0 0.0 
Maryland 66 14.7 125 27.8 223 49.7 35 7.8 
Massachusetts 20 11.6 62 35.8 86 49.7 5 2.9 
Mississipp1 14 25.9 19 35.2 17 31.5 4 74 
New Jersey 1 33.3 0 0.0 2 66.7 0 0.0 
New York 1 33.3 0 0.0 1 33.3 1 33.3 
Ohio 23 15.1 49 32.2 68 44.7 12 7.9 
Rhode Island 8 15.1 15 28.3 29 54.7 1 1.9 
Tennessee 9 16.4 21 38.2 23 41.8 2 3.6 
All States 275 16.6 555 33.5 729 44.0 99 6.0 


Note. Numbers and percentages are based on the total number of valid responses (missing = 29). 


Table 12. TA Survey: Q4. The instructions I read to the students covered all of the information necessary to take the 
test. 


Strongly Disagree Disagree Agree Strongly Agree 
State n % n % n % n % 
Arizona 36 18.5 93 47.7 59 30.3 7 3.6 
Arkansas 28 18.2 46 29.9 72 46.8 8 a2 
Colorado 14 19.7 20 28.2 35 49.3 2 2.8 
D.C. 0 0.0 2 50.0 2 50.0 0 0.0 
Florida 0 0.0 0 0.0 1 100.0 0 0.0 
Illinois 34 11.9 93 32.6 140 49.1 18 6.3 
Louisiana 2 28.6 3 42.9 1 14.3 1 14.3 
Maryland 45 10.1 143 32.0 221 49.4 38 8.5 
Massachusetts 16 9.2 59 33.9 95 54.6 4 2.3 
Mississipp1 10 18.5 17 31.5 23 42.6 4 7.4 
New Jersey Z 66.7 0 0.0 1 33.3 0 0.0 
New York 0 0.0 1 33.3 1 33.3 1 33.3 
Ohio 12 7.9 40 26.3 88 57.9 12 7.9 
Rhode Island 9 16.7 18 33.3 27 50.0 0 0.0 
Tennessee 5 9.3 20 37.0 26 48.1 3 5.6 
All States 214 12.9 556 33.5 793 47.7 98 5.9 


Note. Numbers and percentages are based on the total number of valid responses (missing = 26). 
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Table 13. TA Survey: Q1-4 Scale Means for Test Administration Experience 


State Mean Standard Deviation 
Arizona 2.30 0.60 
Arkansas 2.42 0.68 
Colorado 2.37 0.65 
D.C. 2.94 0.43 
Florida* -- -- 
Illinois 2.56 0.59 
Louisiana 2.29 0.87 
Maryland 2.60 0.66 
Massachusetts 2.56 0.55 
Mississipp1 2.41 0.74 
New Jersey 1.83 1.04 
New York 2.92 1.13 
Ohio 2.61 0.61 
Rhode Island 2.53 0.59 
Tennessee 2.56 0.63 
All States 2.52 0.64 


Note. Mean scale scores computed by taking the mean of items 1-4, which were rated on a 4-point agreement scale 
where | = Strongly Disagree and 4 = Strongly Agree. Overall internal consistency reliability estimate is a = .84. 
“Mean and standard deviation were not calculated for Florida due to sample size (n = 1). 
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Test Environment 


Table 14. TA Survey: Q1. Were there any problems in the physical environment during the test session that might 
have negatively affected student performance? 


Yes No 

State n % n % 

Arizona 8 4.1 188 95.9 
Arkansas 5 3.2 151 96.8 
Colorado 2 2.8 69 97.2 
D.C. 0 0.0 4 100.0 
Florida 0 0.0 1 100.0 
Illinois 29 10.1 258 89.9 
Louisiana 0 0.0 7 100.0 
Maryland 34 7.4 424 92.6 
Massachusetts 1 0.6 172 99.4 
Mississipp1 1 1.9 53 98.1 
New Jersey 0 0.0 3 100.0 
New York 0 0.0 3 100.0 
Ohio 5 3.3 148 96.7 
Rhode Island 2 3:7 52 96.3 
Tennessee 0 0.0 55 100.0 
All States 87 5.2 1591 94.8 


Note. Numbers and percentages are based on the total number of valid responses (missing = 9). 


Table 15. TA Survey: Qla. Please indicate the nature of the problem in the physical 
environment. (Select all that apply “) 


Lighting Problems Other 
State n % n % 
Arizona (n = 10) 0 0.0 7 70.0 
Arkansas (n = 6) 0 0.0 5 83.3 
Colorado (n = 2) 0 0.0 2 100.0 
D.C. (n= 0) -- -- -- -- 
Florida (n = 0) -- -- -- -- 
Tllinois(n = 29) 0 0.0 29 100.0 
Louisiana (n = 0) -- -- -- -- 
Maryland (n = 37) 2 5.4 31 83.8 
Massachusetts (n = 3) 0 0.0 1 33.3 
Mississippi (n = 1) 0 0.0 0 0.0 
New Jersey (n = 0) -- -- -- -- 
New York (n= 0) -- -- -- -- 
Ohio (n= 5) 0 0.0 5 100.0 
Rhode Island (n = 3) 0 0.0 2 66.7 
Tennessee (n = 0) -- -- -- -- 
All States (n = 96) 2 2.1 82 85.4 


Note. Individuals who selected ‘No’ for item 1 (see Table 14) have been excluded from analyses. Individuals with 
missing responses to item | (see note in Table 14) who responded to item 1a were included in analyses. 

“ Percentages reported for “select all that apply” items should be interpreted with caution, as it is not possible to 
differentiate “missing” (i.e., skipped) vs. “not selected” responses. 
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PARCC’s Timing Policy 


Table 16. TA Survey: Q1. Indicate the proportion of students who needed the Additional Time Allowed. 


None Less than '4 Approximately '4 More than 4 
State n % n % n % n % 
Arizona 20 10.2 84 42.9 34 17.3 58 29.6 
Arkansas 55 35.3 67 42.9 15 9.6 19 12.2 
Colorado 18 26.1 36 52:2 2 2.9 13 18.8 
D.C. 0 0.0 0 0.0 0 0.0 3 75.0 
Florida 0 0.0 1 100.0 0 0.0 0 0.0 
Illinois 73 25.8 123 43.5 35 12.4 52 18.4 
Louisiana 1 16.7 3 50.0 0 0.0 2 33.3 
Maryland 145 32.0 190 41.9 67 14.8 51 11.3 
Massachusetts 26 14.9 75 43.1 28 16.1 45 25.9 
Mississippi 14 25.9 27 50.0 8 14.8 5 9.3 
New Jersey 1 33.3 1 33.3 1 33.3 0 0.0 
New York 0 0.0 3 100.0 0 0.0 0 0.0 
Ohio 62 40.8 43 28.3 21 13.8 26 17.1 
Rhode Island wi 12.7 28 50.9 10 18.2 10 18.2 
Tennessee 12 21.8 30 54.5 8 14.5 5 9.1 
All States 434 26.1 713 42.8 230 13.8 289 17.3 


Note. Numbers and percentages are based on the total number of valid responses (missing = 21). 


Table 17. TA Survey: Qla. Please indicate how much additional time was used by the majority of these students. 


Less than 10 minutes Between 10 — 20 minutes More than 20 minutes 
State n % n % n % 
Arizona 17 9.7 95 54.0 64 36.4 
Arkansas 30 29.7 52 51.5 19 18.8 
Colorado 5 9.8 39 76.5 7 13.7 
D.C. 0 0.0 0 0.0 3 75.0 
Florida 0 0.0 1 100.0 0 0.0 
Illinois 46 21.8 113 53.6 52 24.6 
Louisiana 0 0.0 3 60.0 2 40.0 
Maryland 65 21.1 191 62.0 52 16.9 
Massachusetts 8 5.5 93 63.7 45 30.8 
Mississippi 6 15.0 24 60.0 10 25.0 
New Jersey 0 0.0 1 50.0 1 50.0 
New York 1 33.3 2 66.7 0 0.0 
Ohio 12 13.3 64 71.1 14 15.6 
Rhode Island 3 6.3 28 58.3 17 35.4 
Tennessee 10 23.8 2) 64.3 5 11.9 
All States 205 16.7 734 59.7 291 23.7 


Note. Numbers and percentages are based on the total number of valid responses (missing = 23). Individuals who 
selected ‘None’ for item | (see Table 16) have been excluded from analyses. Individuals with missing responses to 
item | (see note in Table 16) who responded to this item were included in analyses. 
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Accessibility Features and Accommodations 


Table 18. TA Survey: Q1. Are you aware that there is an approved PARCC Accessibility Features and 
Accommodations Manual? 


Yes No 

State n % n % 

Arizona 121 61.7 75 38.3 
Arkansas 104 66.7 52 33.3 
Colorado 39 55.7 31 44.3 
D.C. 2 66.7 1 33.3 
Florida 0 0.0 1 100.0 
Illinois 169 59.9 113 40.1 
Louisiana 3 50.0 3 50.0 
Maryland 308 68.3 143 31.7 
Massachusetts 121 69.5 53 30.5 
Mississipp1 34 64.2 19 35.8 
New Jersey 1 33.3 2 66.7 
New York 0 0.0 3 100.0 
Ohio 97 63.8 55 36.2 
Rhode Island 47 85.5 8 14.5 
Tennessee 32 59.3 22 40.7 
All States 1079 64.9 583 35.1 


Note. Numbers and percentages are based on the total number of valid responses (missing = 25). 


Table 19. TA Survey: Qla. Did you read the “Accessibility Features and Accommodations Manual” prior to 
administration ? 


Yes No 

State n % n % 

Arizona 94 78.3 26 21.7 
Arkansas 73 70.9 30 29.1 
Colorado 20 52.6 18 47.4 
D.C. 1 50.0 1 50.0 
Florida 0 0.0 0 0.0 
Illinois 109 64.5 60 35.5 
Louisiana 2 66.7 1 33.3 
Maryland 226 73.1 83 26.9 
Massachusetts 88 Ded 33 27.3 
Mississippi 19 54.3 16 45.7 
New Jersey 1 100.0 0 0.0 
New York 0 0.0 0 0.0 
Ohio 67 69.8 29 30.2 
Rhode Island 34 73.9 12 26.1 
Tennessee 18 56.3 14 43.8 
All States 753 70.0 323 30.0 


Note. Numbers and percentages are based on the total number of valid responses (missing = 28). Individuals who 
selected ‘No’ for item 1 (see Table 18) have been excluded from analyses. Individuals with missing responses to 
item | (see note in Table 18) who responded to item 1a were included in analyses. 
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Table 20. TA Survey: Q2. Indicate whether you administered any of the following 
accessibility features or accommodations to your students during the Field Test 
Administration (select all that apply“). 


Not Applicable 

State n % 

Arizona 95 48.0 
Arkansas 70 44.6 
Colorado 37 52.1 
D.C. 1 25.0 
Florida 0 0.0 

Illinois 162 56.4 
Louisiana 3 42.9 
Maryland 211 45.8 
Massachusetts 69 39.4 
Mississipp1 29 53.7 
New Jersey 1 33.3 
New York 1 33.3 
Ohio 72 47.1 
Rhode Island 30 54.5 
Tennessee 28 50.9 
All States 811 48.1 


Note. * Percentages reported for “select all that apply” items should be interpreted with caution, as it is not possible 
to differentiate “missing” (i.e., skipped) vs. “not selected” responses. 
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Table 21. TA Survey: Q2. Indicate whether you administered any of the following accessibility features or accommodations to your students during the Field 
Test Administration (select all that apply “). 


Clarify Repeat Redirect a Human Reader Human Reader 

Directions Directions Student Math English Scribe English Scribe Math 
State n % n % n % n n % n % n % n 
Arizona (n= 103) 68 66.0 81 78.6 52 50.5 17 16.5 1 1.0 0 0.0 3 2.9 
Arkansas (n = 87) 34 39.1 38 43.7 34 39.1 30 34.5 13 14.9 2 2.3 1 1.1 
Colorado (n = 34) 14 41.2 17 50.0 14 41.2 6 17.6 5 14.7 0 0.0 1 2.9 
D.C. (n = 3) 2 66.7 2 66.7 2 66.7 0 0.0 0 0.0 0 0.0 0 0.0 
Florida (n = 1) 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 1 100.0 0 0.0 
Illinois (n = 125) 74 59.2 70 56.0 56 44.8 29 23.2 16 12.8 6 4.8 4 3.2 
Louisiana (n = 4) 2 50.0 2 50.0 1 25.0 0 0.0 0 0.0 0 0.0 0 0.0 
Maryland (n = 250) 89 35.6 117 46.8 93 37.2 60 24.0 36 14.4 27 10.8 36 14.4 
Massachusetts (n = 106) 75 70.8 58 54.7 62 58.5 24 22.6 6 5.7 5 4.7 3 2.8 
Mississippi (n = 25) 11 44.0 13 52.0 6 24.0 8 32.0 2 8.0 0 0.0 0 0.0 
New Jersey (n = 2) 1 50.0 0 0.0 0 0.0 0 0.0 1 50.0 1 50.0 0 0.0 
New York (n= 2) 1 50.0 1 50.0 2 100.0 0 0.0 0 0.0 0 0.0 0 0.0 
Ohio (n= 81) 40 49.4 40 49.4 26 32.1 34 42.0 11 13.6 2 2 14 17.3 
Rhode Island (n = 25) 11 44.0 14 56.0 15 60.0 9 36.0 1 4.0 1 4.0 0 0.0 
Tennessee (n = 27) 11 40.7 13 48.1 12 44.4 6 22.2 4 14.8 0 0.0 0 0.0 
All States (n = 876) 434 49.5 467 53.3 376 42.9 223 25.5 96 11.0 45 5.1 62 7.1 


Note. Individuals who selected N/A for this item (see Table 20) have been excluded from analyses. 
“ Percentages reported for “select all that apply” items should be interpreted with caution, as it is not possible to differentiate “missing” (i.e., skipped) vs. “not 
selected” responses. 
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Table 22. TA Survey: Q3. If you served as a human reader (read text aloud) to a student 
and/or served as a scribe, please indicate if you read the following documents (select all 


that apply“): 


Not Applicable 

State n % 

Arizona 143 72.2 
Arkansas 97 61.8 
Colorado 49 69.0 
D.C. 3 75.0 
Florida 1 100.0 
Illinois 228 79.4 
Louisiana 6 85.7 
Maryland 314 68.1 
Massachusetts 114 65.1 
Mississipp1 35 64.8 
New Jersey 2 66.7 
New York 3 100.0 
Ohio 97 63.4 
Rhode Island 34 61.8 
Tennessee 42 76.4 
All States 1170 69.4 


Note. * Percentages reported for “select all that apply” items should be interpreted with caution, as it is not possible 
to differentiate “missing” (i.e., skipped) vs. “not selected” responses. 


Table 23. TA Survey: Q3. If you served as a human reader (read text aloud) to a student and/or served as a scribe, 
please indicate if you read the following documents (select all that apply“): 


Appendix B Appendix C Appendix I Appendix J 
State n % n % n % n % 
Arizona (n = 55) 11 20.0 4 7.3 2 3.6 10 18.2 
Arkansas (n = 60) 23 38.3 4 6.7 9 15.0 19 31.7 
Colorado (n = 22) 7 31.8 1 4.5 4 18.2 1 4.5 
D.C. (n= 1) 0 0.0 0 0.0 0 0.0 0 0.0 
Florida (n = 0) -- -- -- -- -- -- -- -- 
Illinois (n = 59) 21 35.6 11 18.6 7 11.9 13 22.0 
Louisiana (n = 1) 0 0.0 0 0.0 0 0.0 0 0.0 
Maryland (n = 147) 51 34.7 45 30.6 27 18.4 39 26.5 
Massachusetts (n = 61) 15 24.6 7 11.5 5 8.2 11 18.0 
Mississippi (n = 19) 9 47.4 0 0.0 0 0.0 2 10.5 
New Jersey (n= 1) 1 100.0 1 100.0 1 100.0 0 0.0 
New York (n = 0) -- -- -- -- -- -- -- -- 
Ohio (n = 56) 34 60.7 14 25.0 4 7.1 17 30.4 
Rhode Island (n = 21) 5 23.8 2 9:5 1 4.8 4 19.0 
Tennessee (n = 5) 5 38.5 1 Tel 4 30.8 3 23.1 
All States (n = 517) 182 35.2 90 17.4 64 12.4 119 23.0 


Note. Individuals who selected N/A for this item (see Table 22) have been excluded from analyses. 
“ Percentages reported for “select all that apply” items should be interpreted with caution, as it is not possible to 
differentiate “missing” (i.e., skipped) vs. “not selected” responses. 
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Table 24. TA Survey: Q4. Did you receive training on administering accessibility features/accommodations from 


your school or district? 


Yes No 

State n % n % 

Arizona 74 40.0 111 60.0 
Arkansas 75 51.7 70 48.3 
Colorado 21 32.3 44 67.7 
D.C. 1 33.3 2 66.7 
Florida 1 100.0 0 0.0 
Illinois 63 22.8 213 77.2 
Louisiana 3 50.0 3 50.0 
Maryland 262 61.1 167 38.9 
Massachusetts 76 46.1 89 53.9 
Mississipp1 22 42.3 30 57.7 
New Jersey 2 66.7 1 33.3 
New York 0 0.0 3 100.0 
Ohio 57 38.5 91 61.5 
Rhode Island 14 25.9 40 74.1 
Tennessee 21 40.4 31 59.6 
All States 693 43.6 896 56.4 


Note. Numbers and percentages are based on the total number of valid responses (missing = 98). 
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Sample Items/Tutorials 


Table 25. TA Survey: Q1. Did the students in your session(s) practice with PARCC sample items prior to 
administration? 


Yes No Not Sure/Don’t Know 
State n % n % n % 
Arizona 49 25.9 114 60.3 26 13.8 
Arkansas 30 20.3 77 52.0 41 2h. 
Colorado 16 24.6 41 63.1 8 12.3 
D.C. 1 33.3 1 33.3 1 33.3 
Florida 1 100.0 0 0.0 0 0.0 
Illinois 65 23.3 162 58.1 52 18.6 
Louisiana 0 0.0 6 100.0 0 0.0 
Maryland 149 34.7 161 37.4 120 27.9 
Massachusetts 31 18.3 104 61.5 34 20.1 
Mississippi 11 21.2 18 34.6 23 44.2 
New Jersey 1 33.3 2 66.7 0 0.0 
New York 2 66.7 1 33.3 0 0.0 
Ohio 18 12.0 86 57.3 46 30.7 
Rhode Island 17 31.5 23 42.6 14 25.9 
Tennessee 19 36.5 25 48.1 8 15.4 
All States 411 25.6 823 51.2 373 23.2 


Note. Numbers and percentages are based on the total number of valid responses (missing = 80). 
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Table 26. TA Survey: Q2. Did the students in your session(s) complete the PARCC tutorial prior to administration? 


Yes No Not Sure/Don’t Know 
State n % n % n % 
Arizona 11 5.8 143 74.9 37 19.4 
Arkansas 11 75 87 59.2 49 33.3 
Colorado 4 6.1 49 74.2 13 19.7 
D.C. 0 0.0 3 100.0 0 0.0 
Florida 0 0.0 1 100.0 0 0.0 
Illinois 11 3.9 206 73.8 62 22.2 
Louisiana 1 16.7 5 83.3 0 0.0 
Maryland 60 13.9 213 49.4 158 36.7 
Massachusetts 6 3.6 120 71.4 42 25.0 
Mississipp1 1 1.9 24 46.2 27 51.9 
New Jersey 1 33.3 1 33.3 1 33.3 
New York 0 0.0 1 33.3 2 66.7 
Ohio 5 3.3 91 60.3 55 36.4 
Rhode Island 5 9.1 36 65.5 14 25.5 
Tennessee 3 5.8 33 63.5 16 30.8 
All States 119 74 1016 63.1 476 29.5 


Note. Numbers and percentages are based on the total number of valid responses (missing = 76). 
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Table 27. TA Survey: Q3. In accordance with your state or local policy, please indicate which of the following options was given to students after the completion 


of testing. 

Sit Quietly Dismissed Quiet Recreational Material 
State n % n % n % 
Arizona 66 34.4 6 3.1 120 62.5 
Arkansas 100 67.1 8 5.4 41 27.5 
Colorado 28 43.1 3 4.6 34 52.3 
D.C. 0 0.0 0 0.0 3 100.0 
Florida 1 100.0 0 0.0 0 0.0 
Illinois 88 31.7 12 4.3 178 64.0 
Louisiana 2 33.3 0 0.0 4 66.7 
Maryland 238 55.9 39 9.2 149 35.0 
Massachusetts 33 19.3 4 2.3 134 78.4 
Mississipp1 30 57.7 0 0.0 22 42.3 
New Jersey 2 66.7 0 0.0 1 33.3 
New York 2 66.7 0 0.0 1 33.3 
Ohio 56 37.3 33 22.0 61 40.7 
Rhode Island 19 34.5 1 1.8 35 63.6 
Tennessee 42 80.8 2 3.8 8 15.4 
All States 708 44.0 108 6.7 793 49.3 
Note. Numbers and percentages are based on the total number of valid responses (missing = 78). 
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Process for Cleaning Up Data for PARCC TA Survey 


e Initial dataset had 8736 respondents, which included responses from Test 
Administrators who administered the computer/tablet-based test (CBT) and Test 
Administrators who administered the paper-based test (PBT). 


Clean-up Steps 
1) Sorted data by start date and deleted cases that began before the initial launch of the 


survey (3/24/14), as well as cases that included words such as ‘test’, ‘delete’, or 
‘dummy’ in the text box items. 
O n=8729 
= CBT respondents = 6478 
= PBT respondents = 2251 


2) Checked for duplicate cases within the dataset; none were found. 
O n=8729 


3) Individuals who only responded to the first five background items (1-4b) and no other 
substantive survey content were removed. 
O n=8583 
=" CBT respondents = 6362 
= PBT respondents = 2221 


4) Further examined survey content items for each assessment type (CBT and PBT) to 
identify participants who failed to respond to any substantive content on the survey. So 
as not to exclude any meaningful responses, a conservative guideline was applied for 
removing cases; only individuals with 2 or fewer responses (not including background 
items) were removed. 

O n=6322 
= CBT respondents = 4635 
= PBT respondents = 1687 


5) Checked for pattern responding. Computed the number of times individuals used a 
particular response option (i.e., response 1, response 2, etc.) to help determine whether 
respondent provided thoughtful responses or was simply selecting the first response 
option for every item on the survey, for example. No evidence of pattern responding 
was found. Consequently, no cases were removed at this step. 

O n=6322 
= CBT respondents = 4635 
= PBT respondents = 1687 
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Appendix K — Item-level Results from the Student Survey for Mathematics: 


Computer/Tablet-based Administration 


Table 1. Student Survey Responses by State (Math CBT) 


Performance-Based Assessment 


End-of-Y ear Assessment 


State n % of Total Responses n % of Total Responses 
Arizona 15,180 8.3 12,615 8.0 
Arkansas 8,283 4.5 7,641 4.8 
Colorado 4,907 2.7 5,480 3.5 

District of Columbia 460 0.3 2,557 1.6 

Florida 2,450 1.3 2,010 1.3 

Illinois 24,746 13.5 24,622 15.6 
Louisiana 9,914 5.4 7,683 49 
Maryland 13,553 7A 13,673 8.7 
Massachusetts 17,163 9.4 16,843 10.7 
Mississippi 13,329 73 10,825 6.9 

New Jersey 22,151 12.1 21,352 13.5 

New Mexico 2,816 1.5 1,977 1.3 

New York 3,243 1.8 3,484 22 

Ohio 26,688 14.6 20,644 13.1 

Rhode Island 4,402 2.4 2,773 1.8 
Tennessee 13,814 75 3,693 2.3 

All States 183,099 100.0 157,872 100.0 
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Table 2. Student Survey Math CBT (PBA): Q1. Did you understand all of the directions read by the person who 
gave you the test? 


Yes No 

State n % n % 

Arizona 12,690 83.7 2,474 16.3 
Arkansas 7,387 89.2 891 10.8 
Colorado 4,154 84.8 TAT 15.2 
District of Columbia 405 88.0 55 12.0 
Florida 2,227 91.0 219 9.0 
Illinois 21,114 85.4 3,616 14.6 
Louisiana 8,910 89.9 1,000 10.1 
Maryland 11,942 88.3 1,582 11.7 
Massachusetts 15,243 88.9 1,909 11.1 
Mississippi 11,534 86.6 1,779 13.4 
New Jersey 19,787 89.4 2,337 10.6 
New Mexico 2,427 86.3 385 13.7 
New York 2,741 84.6 498 15.4 
Ohio 22,354 83.8 4,307 16.2 
Rhode Island 3,796 86.3 601 13.7 
Tennessee 12,269 88.9 1,525 11.1 
All States 158,980 86.9 23,925 13.1 


Note. Percentages are based on the number of valid responses. Missing = 194. 


Table 3. Student Survey Math CBT (EOY): Q1. Did you understand all of the directions read by the person who 
gave you the test? 


Yes No 

State n % n % 

Arizona 9,605 76.2 2,993 23.8 
Arkansas 6,137 80.3 1,503 19.7 
Colorado 4,252 77.7 1,221 22.3 
District of Columbia 2,088 81.7 468 18.3 
Florida 1,651 82.2 357 17.8 
Illinois 18,792 76.4 5,818 23.6 
Louisiana 6,246 81.4 1,426 18.6 
Maryland 10,471 76.7 3,179 23.3 
Massachusetts 13,512 80.3 3,316 19.7 
Mississippi 8,482 78.4 2,332 21.6 
New Jersey 17,196 80.6 4,142 19.4 
New Mexico 1,599 80.9 378 19.1 
New York 2,634 75.7 844 24,3 
Ohio 15,345 74.4 D272 25.6 
Rhode Island 2,113 76.3 655 23.7 
Tennessee 2,825 76.6 861 23.4 
All States 122,948 78.0 34,765 22.0 


Note. Percentages are based on the number of valid responses. Missing = 159. 
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Table 4. Student Survey Math CBT (PBA): Q2. How often was it hard to understand the directions for the questions 
on this test? 


Almost Always Most of the Time Some of the Time Almost Never 
State n % n % n % n % 
Arizona 2,598 17.2 4,277 28.2 6,576 43.4 1,691 11.2 
Arkansas 1,137 13.7 2,036 24.6 3,779 45.7 1,319 15.9 
Colorado 903 18.4 1,362 27.8 2,114 43.2 519 10.6 
District of Columbia 59 12.9 138 30.1 209 45.5 53 11.5 
Florida 290 11.8 611 24.9 1,150 47.0 398 16.3 
Illinois 4,481 18.1 6,654 26.9 10,787 43.7 2,785 11.3 
Louisiana 1,358 13.7 2,580 26.1 4,366 44.1 1,593 16.1 
Maryland 1,857 13,7 3,314 24.5 6,393 47.3 1,950 14.4 
Massachusetts 2,180 12.7 4,274 25.0 8,176 47.7 2,500 14.6 
Mississippi 2,446 18.4 3,569 26.8 5,774 43.4 1,524 11.4 
New Jersey 2,930 13.3 5,629 25.5 10,490 47.4 3,059 13.8 
New Mexico 495 17.6 794 28.3 1,185 42.2 335 11.9 
New York 464 14.4 831 25.7 1,522 47.1 415 12.8 
Ohio 5,676 21.3 7,009 26.3 10,974 41.2 2,986 11.2 
Rhode Island 975 22.2 1,163 26.5 1,757 40.0 496 11.3 
Tennessee 2,343 17.0 3,471 25.2 6,066 44.0 1,907 13.8 
All States 30,192 16.5 47,712 26.1 81,318 44.5 23,530 12.9 


Note. Percentages are based on the number of valid responses. Missing = 347. 


Table 5. Student Survey Math CBT (EOY): Q2. How often was it hard to understand the directions for the questions 
on this test? 


Almost Always Most of the Time Some of the Time Almost Never 
State n % n % n % n % 
Arizona 1,813 14.4 3,147 25.0 5,773 45.9 1,841 14.6 
Arkansas 924 12.1 1,651 21.6 3,528 46.2 1,526 20.0 
Colorado 647 11.8 1,236 22.6 2,664 48.7 925 16.9 
District of Columbia 303 11.9 540 21.2 1,125 44.1 583 22.9 
Florida 234 11.7 418 20.8 925 46.1 428 21.3 
Illinois 3,436 14.0 5,543 22.5 11,486 46.7 4,130 16.8 
Louisiana 989 12.9 1,687 22.0 3,473 45.4 1,509 19.7 
Maryland 1,603 11.8 2,910 21.3 6,552 48.0 2,573 18.9 
Massachusetts 1,975 11.7 3,390 20.2 7,792 46.4 3,654 21.7 
Mississipp1 1,942 18.0 2,545 23.6 4,516 41.8 1,788 16.6 
New Jersey 2,454 11.5 4,412 20.7 10,159 47.7 4,293 20.1 
New Mexico 284 14.4 509 25.8 891 45.2 289 14.6 
New York 393 11.3 733 21.1 1,666 48.1 675 19.5 
Ohio 3,714 18.0 4,718 22.9 8,888 43.2 3,274 15.9 
Rhode Island 488 17.7 673 24.4 1,148 41.6 451 16.3 
Tennessee 625 17.0 814 22.1 1,622 44.1 616 16.8 
All States 21,824 13.9 34,926 22.2 72,208 45.8 28,555 18.1 


Note. Percentages are based on the number of valid responses. Missing = 359. 
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Table 6. Student Survey Math CBT (PBA): Q3. How many questions asked you about things you have not learned 
in school this year? 


All of Them Most of Them Few of Them None of Them 
State n % n % n % n % 
Arizona 820 5.4 3,930 26.0 7,862 52.0 2,507 16.6 
Arkansas 396 4.8 2,002 24.2 4,320 52.2 1,555 18.8 
Colorado 278 5.7 1,586 32.4 2,390 48.8 640 13.1 
District of Columbia 34 74 102 22.2 207 45.1 116 25.3 
Florida 77 3.1 542 22.2 1,370 56.0 457 18.7 
Illinois 1,242 5.0 8,193 33.2 12,340 50.0 2,920 11.8 
Louisiana 457 4.6 2,415 24.4 5,113 51.7 1,911 19.3 
Maryland 726 5.4 3,539 26.2 7,162 53.0 2,084 15.4 
Massachusetts 694 4.1 4,486 26.2 9,036 52.8 2,906 17.0 
Mississippi 715 5.4 3,511 26.4 6,549 49.2 2,539 19.1 
New Jersey 1,012 4.6 6,890 31.2 11,330 51.2 2,883 13.0 
New Mexico 173 6.2 826 29.4 1,349 48.0 461 16.4 
New York 164 5.1 832 25.8 1,621 50.2 610 18.9 
Ohio 1,932 73 9,173 34.5 12,500 47.0 3,003 11.3 
Rhode Island 308 7.0 1,465 33.4 2,083 47.5 532 12.1 
Tennessee 697 5.1 3,454 25.1 6,933 50.3 2,689 19.5 
All States 9,725 5.3 52,946 29.0 92,165 50.5 27,813 15.2 


Note. Percentages are based on the number of valid responses. Missing = 450. 


Table 7. Student Survey Math CBT (EOY): Q3. How many questions asked you about things you have not learned 
in school this year? 


All of Them Most of Them Few of Them None of Them 
State n % n % n % n % 
Arizona 689 55 3,083 24.5 6,409 51.0 2,390 19.0 
Arkansas 344 45 1,806 23.7 4,050 53.1 1,426 18.7 
Colorado 228 4.2 1,384 24.3 3,018 55.2 840 15.4 
District of Columbia 142 5.6 571 22:5 1,046 41.2 782 30.8 
Florida 59 2.9 269 13.4 1,090 54.3 589 29.3 
Illinois 1,193 49 6,857 27.9 12,883 52.4 3,653 14.9 
Louisiana 440 D5 1,603 20.9 3,864 50.5 1,746 22.8 
Maryland 646 4.7 3,062 22:5 7,588 5527 2,336 17.1 
Massachusetts 780 4.6 3,436 20.5 8,442 50.3 4,134 24.6 
Mississippi 718 6.7 2,637 24.5 4,988 46.3 2,439 22.6 
New Jersey 951 4.5 5,499 25.8 10,395 48.8 4,459 20.9 
New Mexico 112 5.7 574 29.1 942 47.7 345 17.5 
New York 172 5.0 640 18.5 1,728 50.0 919 26.6 
Ohio 1,579 7.7 6,178 30.0 9,398 45.7 3,414 16.6 
Rhode Island 164 5.9 802 29.0 1,388 50.3 408 14.8 
Tennessee 212 5.8 844 23.0 1,803 49.1 813 22.1 
All States 8,429 5.4 39,245 24.9 79,032 50.2 30,693 19.5 


Note. Percentages are based on the number of valid responses. Missing = 473. 
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Table 8. Student Survey Math CBT (PBA): Q4. How difficult was this test? 


Easier than School Work Same as School Work Harder than School Work 

State n % n % n % 

Arizona 833 35 4,852 32.1 9,418 62.4 
Arkansas 396 4.8 2,791 33.7 5,086 61.5 
Colorado 216 4.4 1,491 30.5 3,187 65.1 
District of Columbia 50 10.9 207 45.1 202 44.0 
Florida 108 4.4 807 32.9 1,535 62.7 
Illinois 846 3.4 6,007 24.3 17,837 722 
Louisiana 596 6.0 3,392 34.3 5,900 59.7 
Maryland 835 6.2 4,771 35.3 7,909 58.5 
Massachusetts 871 5.1 5,730 33.5 10,516 61.4 
Mississippi 485 3.6 3,812 28.6 9,011 67.7 
New Jersey 690 3.1 6,408 29.0 15,010 67.9 
New Mexico 139 4.9 943 33.6 1,727 61.5 
New York 173 5.4 1,087 33.7 1,970 61.0 
Ohio 852 3.2 6,058 22.8 19,701 74.0 
Rhode Island 113 2.6 913 20.8 3,365 76.6 
Tennessee 578 4.2 3,742 212 9,452 68.6 
All States 7,781 43 53,011 29.0 121,826 66.7 


Note. Percentages are based on the number of valid responses. Missing = 481. 


Table 9. Student Survey Math CBT (EOY): Q4. How difficult was this test? 


Easier than School Work Same as School Work Harder than School Work 

State n % n % n % 

Arizona 894 7A 4,586 36.5 7,070 56.3 
Arkansas 497 6.5 2,684 35.2 4,445 58.3 
Colorado 335 6.1 2,052 37.5 3,079 56.3 
District of Columbia 266 10.5 1,219 47.9 1,058 41.6 
Florida 138 6.9 859 42.9 1,006 50.2 
Illinois 1,329 5.4 7,647 31.1 15,610 63.5 
Louisiana 586 7.7 2,984 39.0 4,088 53.4 
Maryland 1,074 7.9 5,359 39.3 7,201 52.8 
Massachusetts 1,313 78 6,436 38.3 9,038 53.8 
Mississippi 649 6.0 3,477 32.3 6,650 61.7 
New Jersey 1,253 5.9 7,564 35.5 12,480 58.6 
New Mexico 96 4.9 729 37.0 1,146 58.1 
New York 286 8.3 1,500 43.4 1,671 48.3 
Ohio 1,096 5.3 5,931 28.8 13,538 65.8 
Rhode Island 130 4.7 767 27.8 1,863 67.5 
Tennessee 224 6.1 1,235 33.6 2,216 60.3 
All States 10,166 6.5 55,029 35.0 92,159 58.6 


Note. Percentages are based on the number of valid responses. Missing = 518. 
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Table 10. Student Survey Math CBT (PBA): Q5. Did you have enough time to finish this test? 


Finished Very Early _ Finished On Time Rushed to Finish Did Not Finish 
State n % n % n % n % 
Arizona 3,814 25.2 8,985 59.4 1,453 9.6 875 5.8 
Arkansas 2,593 31.4 4,637 56.1 746 9.0 289 3.5 
Colorado 1,458 29.8 2,657 54.3 530 10.8 247 5.0 
District of Columbia 103 22.5 309 67.5 23 5.0 23 5.0 
Florida 559 22.9 1,475 60.3 272 11.1 140 5.7 
Illinois 7,723 31.3 13,891 56.3 2,186 8.9 867 3.5 
Louisiana 2,644 26.7 6,008 60.7 826 8.4 412 4.2 
Maryland 4,545 33.7 6,927 51.3 1,352 10.0 682 5.0 
Massachusetts 4,405 25.8 9,767 57.1 1,707 10.0 1,227 7.2 
Mississipp1 3,486 26.2 8,512 64.0 814 6.1 49] 3.7 
New Jersey 6,250 28.3 12,218 55.4 2,424 11.0 1,181 5.4 
New Mexico 568 20.2 1,752 62.5 310 11.1 175 6.2 
New York 1,059 32.8 1,720 53.3 265 8.2 182 5.6 
Ohio 7,948 29.9 14,763 55.5 2,662 10.0 1,232 4.6 
Rhode Island 1,209 27.6 2,602 59.3 405 9.2 170 3.9 
Tennessee 3,157 27.3 8,060 58.6 1,255 9.1 690 5.0 
All States 52,121 28.6 104,283 57.1 17,230 9.4 8,883 4.9 


Note. Percentages are based on the number of valid responses. Missing = 582. 


Table 11. Student Survey Math CBT (EOY): Q5. Did you have enough time to finish this test? 


Finished Very Early _ Finished On Time Rushed to Finish Did Not Finish 
State n % n % n % n % 
Arizona 3,354 26.7 7,501 59.7 13275 10.2 424 3.4 
Arkansas 2,781 36.5 4,074 53.4 608 8.0 165 2.2 
Colorado 1,870 34.2 3,012 55.1 435 8.0 146 2.7 
District of Columbia 643 25.4 1,630 64.3 165 6.5 97 3.8 
Florida 541 27.0 1,230 61.3 189 9.4 47 2.3 
Illinois 8,942 36.4 13,567 55.2 1,671 6.8 392 1.6 
Louisiana 1,927 25.2 4,870 63.6 651 8.5 210 2.7 
Maryland 5,575 40.9 6,841 50.1 944 6.9 285 2.1 
Massachusetts 5,525 32.9 9,539 56.9 1,328 7.9 383 2.3 
Mississippi 3,101 28.8 6,672 61.9 746 6.9 256 2.4 
New Jersey 7,593 35.7 11,195 52.6 2,002 9.4 503 2.4 
New Mexico 404 20.5 1,263 64.0 242 12.3 63 3.2 
New York 1,198 34.7 1,789 51.8 392 11.3 76 2.2 
Ohio 6,559 31.9 11,462 55.8 2,052 10.0 484 2.4 
Rhode Island 884 32.0 1,672 60.5 166 6.0 42 1.5 
Tennessee 1,121 30.6 2,045 55.8 324 8.8 178 4.9 
All States 52,018 33.1 88,362 56.2 13,190 8.4 3,751 2.4 


Note. Percentages are based on the number of valid responses. Missing = 551. 
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Table 12. Student Survey Math CBT (PBA): Q6. How often do you use a computer or tablet at home? 


A Couple Times A 
Every Day A Few Times a Week Month or Less Do Not Have at Home 

State n % n % n % n % 

Arizona 7,007 46.4 4,603 30.5 1,894 12.5 1,600 10.6 
Arkansas 3,951 47.8 2,257 27.3 1,045 12.6 1,021 12.3 
Colorado 2,530 51.8 1,453 29.7 554 11.3 351 7.2 
District of Columbia 204 44.5 143 31.2 63 13.8 48 10.5 
Florida 1,222 49.9 813 33.2 276 11.3 137 5.6 
Illinois 13,898 56.3 6,987 28.3 2,361 9.6 1,421 5.8 
Louisiana 4,842 49.0 2,870 29.0 1,267 12.8 909 9.2 
Maryland 6,933 51.4 4,277 31.7 1,512 11.2 771 5.7 
Massachusetts 9,567 55.9 5,197 30.4 1,619 95 725 4.2 
Mississipp1 5,932 44.6 3,992 30.0 1,910 14.4 1,462 11.0 
New Jersey 13,305 60.2 6,228 28.2 1,764 8.0 800 3.6 
New Mexico 1,170 41.8 890 31.8 436 15.6 304 10.9 
New York 1,715 53.3 974 30.2 339 10.5 192 6.0 
Ohio 14,752 55.5 7,124 26.8 2,874 10.8 1,831 6.9 
Rhode Island 2,371 54.1 1,438 32.8 380 8.7 191 4.4 
Tennessee 7,003 50.9 3,786 27.5 1,613 11.7 1,358 9.9 
All States 96,402 52.8 53,032 29.1 19,907 10.9 13,121 12 


Note. Percentages are based on the number of valid responses. Missing = 637. 


Table 13. Student Survey Math CBT (EOY): Q6. How often do you use a computer or tablet at home? 


A Couple Times A 
Every Day A Few Times a Week Month or Less Do Not Have at Home 

State n % n % n % n % 

Arizona 5,771 46.0 3,859 30.8 1,706 13.6 1,207 9.6 
Arkansas 3,661 48.0 2,168 28.4 994 13.0 800 10.5 
Colorado 2,607 47.7 1,812 33.2 682 12.5 359 6.6 
District of Columbia 1,094 43.2 899 35.5 333 13.1 208 8.2 
Florida 1,023 51.0 623 31.0 250 12.5 111 5.5 
Illinois 13,380 54.5 7,251 29.5 2,596 10.6 1,343 5.5 
Louisiana 3,667 47.9 2,236 29.2 1,085 14.2 667 8.7 
Maryland 6,918 50.7 4,327 31.7 1,707 12.5 686 5.0 
Massachusetts 9,292 55.4 5,147 30.7 1,697 10.1 635 3.8 
Mississipp1 4,757 44.1 3,203 29.7 1,709 15.9 1,109 10.3 
New Jersey 12,468 58.6 6,147 28.9 1,925 9.0 749 3:5 
New Mexico 818 41.5 649 33.0 284 14.4 218 11.1 
New York 1,699 49.2 1,131 32.8 404 11.7 217 6.3 
Ohio 11,050 53.8 5,635 215 2,343 11.4 1,497 73 
Rhode Island 1,477 53.6 892 32.3 286 10.4 103 3i7, 
Tennessee 1,757 48.1 1,017 27.8 523 14.3 359 9.8 
All States 81,439 51.8 46,996 29.9 18,524 11.8 10,268 6.5 


Note. Percentages are based on the number of valid responses. Missing = 645. 
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Table 14. Student Survey Math CBT (PBA): Q7. How often do you use a computer or tablet in school? 


A Couple Times A 
Every Day A Few Times a Week Month or Less Do Not Use in School 

State n % n % n % n % 

Arizona 2,741 18.2 5,682 37.6 5,394 35.7 1,284 8.5 
Arkansas 2,343 28.3 3,528 42.7 2,123 25.7 276 3.3 
Colorado 1,287 26.3 2,181 44.6 1,308 26.8 111 2.3 
District of Columbia 116 25.5 169 37.1 141 31.0 29 6.4 
Florida 385 15.8 852 34.9 1,014 41.6 188 7.7 
Illinois 6,513 26.4 10,806 43.8 6,702 27.2 633 2.6 
Louisiana 2,421 24.5 3,586 36.3 3,286 33.3 583 5.9 
Maryland 1,866 13.8 5,404 40.1 5,549 41.2 662 49 
Massachusetts 3,001 17.6 7,162 41.9 6,157 36.0 769 4.5 
Mississippi 3,650 27.5 5,140 38.7 3,632 27.3 869 6.5 
New Jersey 4,248 19.3 9,572 43.4 7,565 34.3 679 3.1 
New Mexico 548 19.6 1,065 38.1 1,047 37.4 138 49 
New York 580 18.1 1,321 41.1 1,060 33.0 252 7.8 
Ohio 7,115 26.8 10,667 40.2 8,032 30.2 743 2.8 
Rhode Island 592 13.5 2,060 47.0 1,545 35.3 183 4.2 
Tennessee 2,446 17.8 5,662 41.2 5,134 37.3 504 3.7 
All States 39,852 21.9 74,857 41.1 59,689 32.7 7,903 4.3 


Note. Percentages are based on the number of valid responses. Missing = 798. 


Table 15. Student Survey Math CBT (EOY): Q7. How often do you use a computer or tablet in school? 


A Couple Times A 
Every Day A Few Times a Week Month or Less Do Not Use in School 

State n % n % n % n % 

Arizona 2,378 19.0 5,039 40.2 4,135 33.0 973 7.8 
Arkansas 2,204 28.9 3,223 42.3 1,972 25.9 225 3.0 
Colorado 1,155 21.2 2,400 44.0 1,733 31.8 167 3.1 
District of Columbia 537 21.2 1,122 44.2 773 30.5 105 4.1 
Florida 351 17.6 709 35.5 803 40.2 135 6.8 
Illinois 6,532 26.6 10,925 44.5 6,420 26.2 669 2.7 
Louisiana 1,713 22.4 2,768 36.2 2,735 35.8 429 5.6 
Maryland 1,886 13.8 5,391 39.6 5,761 42.3 584 43 
Massachusetts 3,216 19.2 7,013 41.9 5,919 35.3 601 3.6 
Mississippi 2,986 27.7 3,964 36.8 3,144 29.2 672 6.2 
New Jersey 4,241 19.9 9,338 43.9 7,051 33.1 641 3.0 
New Mexico 495 25.2 651 33.1 726 36.9 94 4.8 
New York 659 19.1 1,390 40.4 1,148 33.4 245 7.1 
Ohio 5,228 25.5 8,090 39.5 6,425 31.3 758 3.7 
Rhode Island 421 15.3 1,247 45.3 955 34.7 130 4.7 
Tennessee 684 18.7 1,412 38.6 1,376 37.7 182 5.0 
All States 34,686 22.1 64,682 41.2 51,076 32.5 6,610 4.2 


Note. Percentages are based on the number of valid responses. Missing = 818. 
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Table 16. Student Survey Math CBT (PBA): Q8. When writing a story or essay, how often do you use a computer or 
tablet? 


All of the Time Most of the Time Some of the Time Never 

State n % n % n % n % 

Arizona 1,774 11.8 3,649 24.2 5,912 39.2 3,745 24.8 
Arkansas 816 9.9 1,719 20.8 3,664 44,3 2,070 25.0 
Colorado 7717 15.9 1,605 32.9 1,926 39.4 577 11.8 
District of Columbia 57 12.5 104 22.8 183 40.0 113 24.7 
Florida 227 9.3 453 18.5 919 37.6 844 34.5 
Illinois 4,619 18.7 7,674 31.1 8,928 36.2 3,433 13.9 
Louisiana 1,189 12.0 2,101 21.3 3,849 39.0 2,739 27.7 
Maryland 1,840 13.6 3,488 25.9 5,562 41.3 2,592 19.2 
Massachusetts 2,857 16.7 4,806 28.1 6,325 37.0 3,096 18.1 
Mississippi 1,637 12.3 3,295 24.8 5,330 40.1 3,021 22.7 
New Jersey 3,533 16.0 6,845 31.0 8,510 38.5 3,198 14.5 
New Mexico 396 14.1 726 25.9 1,149 41.0 532 19.0 
New York 436 13.6 789 24.5 1,246 38.7 746 23.2 
Ohio 4,013 15.1 7,776 29.3 10,637 40.0 4,137 15.6 
Rhode Island 844 19,3 1,322 30.2 1,638 37.5 569 13.0 
Tennessee 1,463 10.6 3,643 26.5 6,104 44.4 2,531 18.4 
All States 26,478 14.5 49,995 27.4 71,882 39.4 33,943 18.6 


Note. Percentages are based on the number of valid responses. Missing = 801. 


Table 17. Student Survey Math CBT (EOY): Q8. When writing a story or essay, how often do you use a computer or 
tablet? 


All of the Time Most of the Time Some of the Time Never 

State n % n % n % n % 

Arizona 1,694 13.5 3,185 25.4 4,885 39.0 2,759 22.0 
Arkansas 791 10.4 1,797 23.6 3,388 44.4 1,647 21.6 
Colorado 741 13.6 1,659 30.4 2,245 41.1 811 14.9 
District of Columbia 383 15.1 663 26.1 1,046 41.2 447 17.6 
Florida 175 8.8 348 17.4 768 38.4 708 35.4 
Illinois 4,222 17.2 7,933 32.3 8,664 35.3 3,738 15.2 
Louisiana 977 12.8 1,636 21.4 3,009 39.3 2,025 26.5 
Maryland 1,887 13.9 3,483 25.6 5,655 41.5 2,593 19.0 
Massachusetts 2,826 16.9 4,739 28.3 6,233 37.2 2,971 17.7 
Mississippi 1,557 14.5 2,707 25.2 3,975 36.9 2,520 23.4 
New Jersey 3,343 15.7 6,356 29.9 8,397 39.5 3,185 15.0 
New Mexico 284 14.4 585 29.7 788 40.0 312 15.8 
New York All 11.9 898 26.1 1,402 40.7 733 21.3 
Ohio 3,560 17.4 5,939 29.0 7,715 37.6 3,285 16.0 
Rhode Island 559 20.3 859 31.2 950 34.5 389 14.1 
Tennessee 444 12.1 846 23.1 1,632 44.5 742 20.3 
All States 23,854 15.2 43,633 27.8 60,752 38.7 28,865 18.4 


Note. Percentages are based on the number of valid responses. Missing = 768. 
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Table 18. Student Survey Math CBT (PBA): Q9. Would you rather take this test on paper OR on a computer or 
tablet? 


On Paper On a Computer or Tablet 
State n % n % 
Arizona 5,462 36.2 9,628 63.8 
Arkansas 3,688 44.6 4,581 55.4 
Colorado 2,344 48.0 2,541 52.0 
District of Columbia 103 22.5 355 Th 
Florida 858 35.1 1,585 64.9 
Illinois 10,365 42.1 14,274 57.9 
Louisiana 3,409 34.5 6,472 65.5 
Maryland 3,782 28.0 9,716 72.0 
Massachusetts 7,596 44.5 9,490 55.5 
Mississipp1 4,896 36.9 8,389 63.1 
New Jersey 10,819 49.0 11,259 51.0 
New Mexico 1,040 37.1 1,763 62.9 
New York 1,427 44.3 1,791 55.7 
Ohio 14,206 53.5 12,363 46.5 
Rhode Island 1,807 41.3 2,570 58.7 
Tennessee 4,986 36.3 8,751 63.7 
All States 76,788 42.1 105,528 57.9 


Note. Percentages are based on the number of valid responses. Missing = 783. 


Table 19. Student Survey Math CBT (EOY): Q9. Would you rather take this test on paper OR on a computer or 
tablet? 


On Paper On a Computer or Tablet 
State n % n % 
Arizona 3,972 31.7 8,557 68.3 
Arkansas 3,011 39.5 4,613 60.5 
Colorado 1,962 36.0 3,493 64.0 
District of Columbia 569 22.4 1,966 717.6 
Florida 671 33.4 1,337 66.6 
Illinois 8,431 34.3 16,116 65.7 
Louisiana 2,444 32.0 5,205 68.0 
Maryland 3,346 24.5 10,292 13D 
Massachusetts 6,849 40.8 9,924 59.2 
Mississipp1 4,412 41.0 6,351 59.0 
New Jersey 9,384 44.1 11,896 55.9 
New Mexico 634 32.2 1,332 67.8 
New York 1,318 38.4 2,118 61.6 
Ohio 10,208 49.8 10,305 50.2 
Rhode Island 921 33.4 1,836 66.6 
Tennessee 1,279 35.0 2,376 65.0 
All States 59,411 37.8 97,717 62.2 


Note. Percentages are based on the number of valid responses. Missing = 744. 
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Table 20. Student Survey Math CBT (PBA): Q10. What did you use to take this test? 


Desktop Computer Laptop Computer Tablet with a Keyboard Tablet without Keyboard 

State n % n % n % n % 

Arizona 9,990 66.2 4,228 28.0 652 4.3 215 1.4 
Arkansas 5,007 60.6 2,927 35.4 276 3.3 55 0.7 
Colorado 2,796 57.2 1,953 40.0 102 2.1 36 0.7 
District of Columbia 263 57.4 155 33.8 31 6.8 9 2.0 
Florida 2,253 92.3 103 4.2 67 2.7 18 0.7 
Illinois 15,735 63.8 7,530 30.5 1,082 4.4 306 1.2 
Louisiana 7,488 75.8 1,856 18.8 401 4.1 140 1.4 
Maryland 10,019 74.2 3,062 22.7 327 2.4 88 0.7 
Massachusetts 9,097 53.2 5,875 34.4 1,776 10.4 352 2.1 
Mississippi 11,333 85.3 1,410 10.6 442 3.3 104 0.8 
New Jersey 12,312 55.7 8,373 37.9 1,262 5.7 140 0.6 
New Mexico 2,266 80.8 456 16.3 67 2.4 15 0.5 
New York 2,037 63.3 1,011 31.4 140 4.4 28 0.9 
Ohio 15,727 59.4 9,489 35.8 967 3.7 293 1.1 
Rhode Island 2,304 52.6 1,945 44.4 115 2.6 19 0.4 
Tennessee 8,683 63.2 4,773 34.7 232 1.7 57 0.4 
All States 117,310 64.4 55,146 30.3 7,939 4.4 1,875 1.0 


Note. Percentages are based on the number of valid responses. Missing = 829. 
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Table 21. Student Survey Math CBT (EOY): Q10. What did you use to take this test? 


Desktop Computer Laptop Computer Tablet with a Keyboard Tablet without Keyboard 

State n % n % n % n % 

Arizona 7,977 63.7 3,735 29.8 643 5.1 163 1.3 
Arkansas 4,394 57.6 2,779 36.4 374 4.9 78 1.0 
Colorado 3,141 57.6 1,992 36.5 293 5.4 30 0.5 
District of Columbia 843 33.3 1,589 62.7 79 3.1 24 0.9 
Florida 1,869 93.2 80 4.0 4] 2.0 15 0.7 
Illinois 14,476 59.0 8,781 35.8 941 3.8 351 1.4 
Louisiana 5,681 74.2 1,409 18.4 419 5.5 149 1.9 
Maryland 10,204 74.9 3,021 22.2 308 2.3 96 0.7 
Massachusetts 8,866 52.9 5,529 33.0 1,915 11.4 454 27] 
Mississipp1 8,329 77.4 1,889 17.6 424 3.9 119 1.1 
New Jersey 11,335 53.2 8,508 40.0 1,213 5.7 231 1.1 
New Mexico 1,409 71.6 403 20.5 137 7.0 20 1.0 
New York 2,343 68.1 904 26.3 142 4.1 50 1.5 
Ohio 12,476 60.9 7,028 34.3 803 3.9 185 0.9 
Rhode Island 1,041 37.9 1,593 57.9 97 3.5 18 0.7 
Tennessee 2,270 62.1 1,261 34.5 77 2.1 46 1.3 
All States 96,654 61.5 50,501 32.1 7,906 5.0 2,029 1.3 


Note. Percentages are based on the number of valid responses. Missing = 782. 


Updated February 17, 2015 Page K-12 


PARCC Quality of Test Administration 


Table 22. Student Survey Math CBT (PBA): Q11. How many times did you practice on a computer or tablet to get 
ready for this test? 


Never Once More than Once 

State n % n % n % 

Arizona 7,061 46.9 5,602 37.2 2,398 15.9 
Arkansas 2,702 32.7 3,662 44,3 1,903 23.0 
Colorado 2,884 59.0 1,441 29.5 560 11.5 
District of Columbia 170 37.2 143 31.3 144 31.5 
Florida 1,227 50.3 728 29.8 484 19.8 
Illinois 10,652 43.2 9,929 40.3 4,062 16.5 
Louisiana 4,590 46.4 3,325 33.6 1,967 19.9 
Maryland 4,430 32.8 5,301 39.3 3,756 27.8 
Massachusetts 6,077 35.6 7,516 44.0 3,500 20.5 
Mississippi 6,254 47.1 4,469 33.7 2,555 19.2 
New Jersey 7,833 35.5 9,324 42.2 4,912 22.3 
New Mexico 1,367 48.7 992 35.3 449 16.0 
New York 1,549 48.2 1,088 33.9 574 17.9 
Ohio 14,592 55.1 8,845 33.4 3,051 11.5 
Rhode Island 1,677 38.3 1,756 40.1 946 21.6 
Tennessee 7,221 52.6 4,463 32.5 2,050 14.9 
All States 80,286 44.1 68,584 37.6 33,311 18.3 


Note. Percentages are based on the number of valid responses. Missing = 918. 


Table 23. Student Survey Math CBT (EOY): Q11. How many times did you practice on a computer or tablet to get 
ready for this test? 


Never Once More than Once 
State n % n % n % 
Arizona 5,861 46.8 4,361 34.9 2,291 18.3 
Arkansas 2,410 31.6 3,424 44.9 1,785 23.4 
Colorado 2,756 50.5 1,806 33.1 893 16.4 
District of Columbia 1,061 41.8 818 32.3 657 25.9 
Florida 804 40.1 675 33.7 525 26.2 
Illinois 10,868 44,3 9,166 37.3 4,513 18.4 
Louisiana 3,712 48.5 2,329 30.5 1,605 21.0 
Maryland 5,010 36.8 4,930 36.2 3,688 27.1 
Massachusetts 6,536 39.0 6,596 39.4 3,622 21.6 
Mississippi 5,227 48.6 3,169 29.5 2,356 21.9 
New Jersey 8,021 37.7 8,098 38.1 5,153 24.2 
New Mexico 967 49.1 646 32.8 355 18.0 
New York 1,554 45.1 1,194 34.7 694 20.2 
Ohio 10,280 50.2 6,733 32.9 3,476 17.0 
Rhode Island 999 36.3 1,027 37.3 729 26.5 
Tennessee 2,073 56.7 1,018 27.9 563 15.4 
All States 68,139 43.4 55,990 35.7 32,905 21.0 


Note. Percentages are based on the number of valid responses. Missing = 838. 
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Table 24, Student Survey Math CBT (PBA): Q12. Was it easy to use the highlighter ? 


Yes No Did Not Use Highlighter 

State n % n % n % 

Arizona 6,100 40.5 837 5.6 8,117 53.9 
Arkansas 2,835 34.3 395 4.8 5,037 60.9 
Colorado 1,347 27.6 403 8.3 3,133 64.2 
District of Columbia 185 40.6 39 8.6 232 50.9 
Florida 991 40.6 117 4.8 1,332 54.6 
Illinois 7,029 28.5 1,228 5.0 16,366 66.5 
Louisiana 3,696 37.4 532 5.4 5,650 57.2 
Maryland 5,098 37.8 748 5.5 7,653 56.7 
Massachusetts 6,278 36.8 987 5.8 9,815 57.5 
Mississippi 4,808 36.2 586 4.4 7,874 59.3 
New Jersey 7,189 32.6 1,235 5.6 13,640 61.8 
New Mexico 1,115 39.8 222 7.9 1,466 52.3 
New York 1,121 34.9 193 6.0 1,900 59.1 
Ohio 7,838 29.6 1,672 6.3 16,995 64.1 
Rhode Island 1,540 35.2 321 73 2,517 57.5 
Tennessee 3,920 28.6 721 5.3 9,088 66.2 
All States 61,090 33.5 10,236 5.6 110,815 60.8 


Note. Percentages are based on the number of valid responses. Missing = 958. 


Table 25. Student Survey Math CBT (EOY): Q12. Was it easy to use the highlighter? 


Yes No Did Not Use Highlighter 

State n % n % n % 

Arizona 5,735 45.9 691 a5 6,078 48.6 
Arkansas 3,042 39.9 335 4.4 4,243 55:7 
Colorado 1,973 36.2 269 4.9 3,212 58.9 
District of Columbia 1,116 44.1 146 5.8 1,268 50.1 
Florida 946 47.2 97 4.8 962 48.0 
Illinois 8,805 35.9 957 3.9 14,784 60.2 
Louisiana 3,120 40.8 408 5.3 4,116 53.8 
Maryland 5,762 42.3 562 4.1 7,302 53.6 
Massachusetts 6,380 38.1 943 5.6 9,429 56.3 
Mississippi 4,309 40.1 601 5.6 5,845 54.3 
New Jersey 7,817 36.7 853 4.0 12,604 59.2 
New Mexico 841 42.7 98 5.0 1,030 52.3 
New York 1,435 41.8 186 5.4 1,815 52.8 
Ohio 7,408 36.2 1,242 6.1 11,838 57.8 
Rhode Island 1,084 39.4 145 5.3 1,521 55.3 
Tennessee 1,224 33.6 213 5.8 2,211 60.6 
All States 60,997 38.9 7,746 4.9 88,258 56.2 


Note. Percentages are based on the number of valid responses. Missing = 871. 
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Table 26. Student Survey Math CBT (PBA): Q13. Was it easy to type your answers? 


Yes No Did Not Type Answers 
State n % n % n % 
Arizona 10,485 69.8 4,287 28.5 260 1.7 
Arkansas 6,012 72.8 2,170 26.3 74 0.9 
Colorado 2,901 59.5 1,914 39.3 60 1.2 
District of Columbia 358 78.2 88 19.2 12 2.6 
Florida 1,695 69.6 708 29.1 34 1.4 
Illinois 15,547 63.2 8,802 35.8 256 1.0 
Louisiana 7,184 72.8 2,530 25.6 156 1.6 
Maryland 8,634 64.0 4,634 34.4 221 1.6 
Massachusetts 9,863 57.8 6,994 41.0 211 1.2 
Mississippi 9,804 74.0 3,299 24.9 151 1.1 
New Jersey 12,771 57.9 9,048 41.0 243 1.1 
New Mexico 1,924 68.7 842 30.1 34 1.2 
New York 2,034 63.4 1,122 35.0 53 1.7 
Ohio 15,962 60.2 10,129 38.2 428 1.6 
Rhode Island 2,632 60.1 1,676 38.3 70 1.6 
Tennessee 9,539 69.6 3,970 29.0 202 1.5 
All States 117,345 64.5 62,213 34.2 2,465 1.4 


Note. Percentages are based on the number of valid responses. Missing = 1076. 


Table 27. Student Survey Math CBT (EOY): Q13. Was it easy to type your answers? 


Yes No Did Not Type Answers 

State n % n % n % 

Arizona 10,008 80.1 2,128 17.0 358 2.9 
Arkansas 6,162 80.9 1,264 16.6 187 2:5 
Colorado 4,021 73.8 1,180 21.6 250 4.6 
District of Columbia 2,061 81.5 355 14.0 112 4.4 
Florida 1,614 80.7 325 16.2 62 3.1 
Illinois 19,238 78.4 4,348 17.7 941 3.8 
Louisiana 6,202 81.1 1,218 15.9 223 2.9 
Maryland 10,289 75.6 2,516 18.5 809 5.9 
Massachusetts 12,511 74.7 3,464 20.7 763 4.6 
Mississippi 8,512 79.2 1,875 17.5 355 3.3 
New Jersey 16,354 77.0 4,226 19.9 666 3.1 
New Mexico 1,559 79.4 336 17.1 68 3.5 
New York 2,779 80.9 543 15.8 112 3.3 
Ohio 14,706 71.9 4,840 23.7 917 4.5 
Rhode Island 2,098 76.2 534 19.4 120 4.4 
Tennessee 2,891 79.2 626 17.2 131 3.6 
All States 121,005 77.1 29,778 19.0 6,074 3.9 


Note. Percentages are based on the number of valid responses. Missing = 1015. 
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Table 28. Student Survey Math CBT (PBA): Q14. Was it easy to make pictures or words bigger or smaller? 


Yes No Did Not Change Size 
State n % n % n % 
Arizona 4,182 27.8 1,940 12.9 8,907 59.3 
Arkansas 1,997 24.2 882 10.7 5,371 65.1 
Colorado 763 15.6 805 16.5 3,308 67.8 
District of Columbia 195 43.0 57 12.6 202 44.5 
Florida 620 25.5 371 15.2 1,445 59.3 
Illinois 5,075 20.6 2,913 11.8 16,628 67.5 
Louisiana 3,005 30.4 1,078 10.9 5,788 58.6 
Maryland 3,200 23.7 1,833 13.6 8,466 62.7 
Massachusetts 3,946 23.1 2,398 14.1 10,712 62.8 
Mississippi 3,553 26.8 1,351 10.2 8,364 63.0 
New Jersey 4,407 20.0 2,769 12.6 14,883 67.5 
New Mexico 766 27.3 510 18.2 1,526 54.5 
New York 762 23.8 453 14.1 1,993 62.1 
Ohio 5,345 20.2 3,751 14.2 17,403 65.7 
Rhode Island 955 21.8 735 16.8 2,683 61.4 
Tennessee 3,180 23.2 1,454 10.6 9,076 66.2 
All States 41,951 23.0 23,300 12.8 116,755 64.1 


Note. Percentages are based on the number of valid responses. Missing = 1093. 


Table 29. Student Survey Math CBT (EOY): Q14. Was it easy to make pictures or words bigger or smaller? 


Yes No Did Not Change Size 
State n % n % n % 
Arizona 3,668 29.4 1,478 11.8 7,343 58.8 
Arkansas 1,939 25.5 688 9.0 4,989 65.5 
Colorado 1,033 19.0 665 12.2 3,748 68.8 
District of Columbia 778 30.8 283 11.2 1,467 58.0 
Florida 454 22.7 194 9.7 1,352 67.6 
Illinois 5,268 21.5 2,260 9.2 17,010 69.3 
Louisiana 2,189 28.6 811 10.6 4,647 60.8 
Maryland 3,071 22.5 1,365 10.0 9,185 67.4 
Massachusetts 3,820 22.8 1,813 10.8 11,098 66.3 
Mississippi 2,827 26.3 1,070 10.0 6,839 63.7 
New Jersey 4,581 21.6 2,089 9.8 14,587 68.6 
New Mexico 555 28.2 209 10.6 1,202 61.1 
New York 862 25.1 390 11.4 2,180 63.5 
Ohio 4,719 23.1 2,517 12.3 13,211 64.6 
Rhode Island 672 24.4 358 13.0 1,719 62.5 
Tennessee 797 21.8 369 10.1 2,482 68.0 
All States 37,233 23.7 16,559 10.6 103,059 65.7 


Note. Percentages are based on the number of valid responses. Missing = 1021. 
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Table 30. Student Survey Math CBT (PBA): Q15. Was it easy to enter math symbols and numbers for your 
answers? 


Did Not Enter Math Symbols 


Yes No or Numbers 

State n % n % n % 

Arizona 10,364 69.0 3,875 25.8 787 5.2 
Arkansas 5,869 71.1 1,987 24.1 394 4.8 
Colorado 3,006 61.6 1,637 33.6 235 4.8 
District of Columbia 343 75.1 96 21.0 18 3.9 
Florida 1,805 74.1 558 22.9 74 3.0 
Illinois 16,580 67.4 6,887 28.0 1,141 4.6 
Louisiana 7,164 72.6 2,179 22.1 521 5.3 
Maryland 9,311 69.1 3,553 26.4 607 4.5 
Massachusetts 10,868 63.8 5,540 32.5 635 3.7 
Mississippi 9,888 74.6 2,775 20.9 589 4.4 
New Jersey 14,245 64.6 6,935 31.5 861 3.9 
New Mexico 1,969 70.4 711 25.4 115 4.1 
New York 2,119 66.2 923 28.8 159 5.0 
Ohio 16,364 61.8 8,639 32.6 1,496 5.6 
Rhode Island 2,832 64.8 1,334 30.5 204 4.7 
Tennessee 9,655 70.5 3,349 24.5 691 5.0 
All States 122,382 67.3 50,978 28.0 8,527 4.7 


Note. Percentages are based on the number of valid responses. Missing = 1212. 


Table 31. Student Survey Math CBT (EOY): Q15. Was it easy to enter math symbols and numbers for your 
answers? 


Did Not Enter Math Symbols 


Yes No or Numbers 

State n % n % n % 

Arizona 8,255 66.2 2,809 22.5 1,409 11.3 
Arkansas 5,139 67.5 1,580 20.8 889 11.7 
Colorado 3,435 63.1 1,462 26.9 547 10.0 
District of Columbia 1,789 70.8 500 19.8 238 9.4 
Florida 1,396 70.0 375 18.8 224 11.2 
Illinois 15,955 65.1 5,378 21.9 3,176 13.0 
Louisiana 5,208 68.3 1,530 20.1 887 11.6 
Maryland 9,335 68.6 2,546 18.7 1,730 12.7 
Massachusetts 10,557 63.1 4,144 24.8 2,024 12.1 
Mississippi 6,923 64.6 2,262 21.1 1,532 14.3 
New Jersey 13,577 63.9 5,195 24.4 2,479 11.7 
New Mexico 1,251 63.7 506 25.8 207 10.5 
New York 2,280 66.6 714 20.9 430 12.6 
Ohio 12,226 59.8 5,448 26.7 2,762 13.5 
Rhode Island 1,803 65.6 591 21.5 354 12.9 
Tennessee 2,293 62.9 851 23.3 502 13.8 
All States 101,422 64.7 35,891 22.9 19,390 12.4 


Note. Percentages are based on the number of valid responses. Missing = 1169. 
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Table 32. Student Survey Math CBT (PBA): Q16. Was it easy to use the calculator? 


Yes No Did Not Use Calculator 
State n % n % n % 
Arizona 8,103 54.0 1,296 8.6 5,609 37.4 
Arkansas 4,512 54.7 573 6.9 3,164 38.4 
Colorado 2,928 60.0 514 10.5 1,434 29.4 
District of Columbia 195 42.6 26 5.7 237 51.7 
Florida 771 31.7 181 7A 1,484 60.9 
Illinois 11,793 47.9 2,517 10.2 10,295 41.8 
Louisiana 5,111 51.8 639 6.5 4,109 41.7 
Maryland 6,776 50.2 811 6.0 5,899 43.7 
Massachusetts 8,159 47.8 1,549 9.1 7,353 43.1 
Mississippi 7,211 54.4 999 75 5,051 38.1 
New Jersey 10,274 46.6 2,422 11.0 9,353 42.4 
New Mexico 1,700 60.7 248 8.9 853 30.5 
New York 1,519 47.4 260 8.1 1,423 44.4 
Ohio 13,120 49.6 4,522 17.1 8,834 33.4 
Rhode Island 2,354 53.9 474 10.9 1,540 35.3 
Tennessee 6,874 50.2 1,503 11.0 5,317 38.8 
All States 91,400 50.3 18,534 10.2 71,955 39.6 


Note. Percentages are based on the number of valid responses. Missing = 1210. 


Table 33. Student Survey Math CBT (EOY): Q16. Was it easy to use the calculator? 


Yes No Did Not Use Calculator 
State n % n % n % 
Arizona 6,169 49.6 1,094 8.8 5,179 41.6 
Arkansas 3,873 50.9 645 8.5 3,087 40.6 
Colorado 3,008 55.2 413 7.6 2,025 37.2 
District of Columbia 927 36.7 144 5.7 1,455 57.6 
Florida 532 26.6 122 6.1 1,349 67.3 
Illinois 10,929 44.6 2,240 9.1 11,341 46.3 
Louisiana 3,619 47.4 553 7.2 3,457 45.3 
Maryland 6,337 46.6 742 De 6,533 48.0 
Massachusetts 7,211 43.1 1,657 9.9 7,848 46.9 
Mississippi 5,523 51.7 891 8.3 4,265 39.9 
New Jersey 9,685 45.6 2,340 11.0 9,224 43.4 
New Mexico 1,101 56.1 163 8.3 698 35.6 
New York 1,560 45.7 298 8.7 1,557 45.6 
Ohio 9,626 47.2 3,205 15.7 7,574 37.1 
Rhode Island 1,322 48.1 239 8.7 1,189 43.2 
Tennessee 1,630 44.7 374 10.3 1,639 45.0 
All States 73,052 46.7 15,120 9.7 68,420 43.7 


Note. Percentages are based on the number of valid responses. Missing = 1280. 
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Table 34. Student Survey Math CBT (PBA): Q17. Did any of the problems listed below happen during your test? (Select all that apply)’. 


Computer Stopped Difficulty Dragging or Difficulty Making Changes No Problems Occurred 
Computer Worked Slowly Moving Things on Screen to Answers During the Test 

State n % n % n % n % n % 

Arizona 2,197 14.5 4,401 29.0 3,269 21.5 5,134 33.8 6,692 44.1 
Arkansas 1,002 12.1 2,256 27.2 1,771 21.4 2,979 36.0 3,745 45.2 
Colorado 863 17.6 2,057 41.9 1,449 29.5 2,439 49.7 1,539 31.4 
District of Columbia 61 13.3 109 23.7 70 15.2 90 19.6 238 51.7 
Florida 531 21.7 847 34.6 512 20.9 902 36.8 880 35.9 
Illinois 3,380 13.7 8,243 33.3 5,728 23.1 10,678 43.2 9,522 38.5 
Louisiana 1,240 12.5 2,330 23.5 1,878 18.9 3,186 32.1 4,864 49.1 
Maryland 2,440 18.0 4,977 36.7 3,114 23.0 5,548 40.9 5,012 37.0 
Massachusetts 2,996 17.5 6,447 37.6 4,413 25.7 8,162 47.6 5,777 33.7 
Mississipp1 1,604 12.0 2,908 21.8 2,242 16.8 3,881 29.1 7,044 52.8 
New Jersey 3,951 17.8 8,730 39.4 5,898 26.6 11,280 50.9 7,006 31.6 
New Mexico 421 15.0 900 32.0 614 21.8 1,114 39.6 1,190 42.3 
New York 432 13.3 910 28.1 731 22.5 1,254 38.7 1,358 41.9 
Ohio 3,539 13.3 9,200 34.5 7,118 26.7 11,888 44.5 9,947 37.3 
Rhode Island 826 18.8 1,745 39.6 1,186 26.9 2,002 45.5 1,554 35.3 
Tennessee 1,933 14.0 4,244 30.7 3,067 22.2 5,155 37.3 5,875 42.5 
All States 27,416 15.0 60,304 32.9 43,060 23.5 75,692 41.3 72,243 39.5 


“ Percentages reported for “select all that apply” should be interpreted with caution, as it is not possible to differentiate “missing” (i.e., skipped) vs. “not selected” 


responses. 
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Table 35. Student Survey Math CBT (EOY): Q17. Did any of the problems listed below happen during your test? (Select all that apply)’. 


Computer Stopped Difficulty Dragging or Difficulty Making Changes No Problems Occurred 
Working Computer Worked Slowly Moving Things on Screen to Answers During the Test 

State n % n % n % n % n % 

Arizona 1,510 12.0 2,615 20.7 2,953 23.4 2,401 19.0 6,978 55.3 
Arkansas 803 10.5 1,680 22.0 1,946 25.5 1,385 18.1 4,224 55.3 
Colorado 613 11.2 1,422 25.9 1,492 27.2 1,309 23.9 2,793 51.0 
District of Columbia 277 10.8 595 23.3 567 22.2 404 15.8 1,399 54.7 
Florida 538 26.8 359 17.9 424 21.1 402 20.0 983 48.9 
Illinois 2,263 9.2 5,168 21.0 5,993 24.3 4,846 19.7 13,859 56.3 
Louisiana 1,004 13.1 1,537 20.0 1,722 22.4 1,324 17.2 4,412 57.4 
Maryland 1,830 13.4 3,541 25.9 3,288 24.0 2,873 21.0 7,158 52.4 
Massachusetts 1,994 11.8 4,077 24.2 4,691 27.9 3,843 22.8 8,690 51.6 
Mississipp1 951 8.8 1,789 16.5 2,299 21.2 1,685 15.6 6,528 60.3 
New Jersey 2,413 11.3 5,027 23.5 5,890 27.6 5,004 23.4 10,956 51.3 
New Mexico 264 13.4 470 23.8 488 24.7 386 19.5 1,034 52.3 
New York 375 10.8 592 17.0 829 23.8 610 17.5 1,976 56.7 
Ohio 2,258 10.9 4,839 23.4 6,120 29.6 4,897 23.7 10,435 50.5 
Rhode Island 451 16.3 942 34.0 739 36.6 621 22.4 1,271 45.8 
Tennessee 582 15.8 967 26.2 1,100 29.8 737 20.0 1,798 48.7 
All States 18,126 11.5 35,620 22.6 40541 25.7 32,727 20.7 84,494 53.5 


“ Percentages reported for “select all that apply” should be interpreted with caution, as it is not possible to differentiate “missing” (i.e., skipped) vs. “not selected” 


responses 
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Appendix L — Item-level Results from the Student Survey for Mathematics: 


Paper-based Test Administration 


Table 1. Student Survey Responses by State (Math PBT) 


Performance-Based Assessment 


End-of-Year Assessment 


State n % of Total Responses n % of Total Responses 
Arizona 9,517 13.3 9,474 15.8 
Arkansas 5,027 7.0 4,258 7.1 
Colorado 1,562 2.2 1,948 3.2 

District of Columbia 169 0.2 88 0.1 

Illinois 12,026 16.8 9,515 15.8 
Louisiana 219 0.3 472 0.8 
Maryland 7,338 10.3 7,546 12.6 
Massachusetts 7,040 9.9 7,013 11.7 
Mississippi 7,652 10.7 6,898 11.5 

New Jersey -- -- 6 0.0 

New York 1,642 2.3 1,631 2.7 

Ohio 11,000 15.4 7,481 12.5 

Rhode Island 2,050 2.9 1,407 2.3 
Tennessee 6,210 8.7 2,336 3.9 

All States 71,452 100.0 60,073 100.0 
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Table 2. Student Survey Math PBT (PBA): Q1. Did you understand all of the directions read by the person who 


gave you the test? 


Yes No 

State n % n % 

Arizona 7,756 81.8 1,731 18.2 
Arkansas 4,378 87.2 641 12.8 
Colorado 1,320 84.7 238 15.3 
District of Columbia 141 83.9 27 16.1 
Illinois 10,427 86.9 1,570 13.1 
Louisiana 197 90.0 22 10.0 
Maryland 6,409 87.7 903 12.3 
Massachusetts 6,153 87.7 860 12.3 
Mississipp1 6,695 87.9 922 12.1 
New York 1,368 84.1 259 15.9 
Ohio 9,812 89.6 1,143 10.4 
Rhode Island 1,853 90.6 192 9.4 
Tennessee 5,547 89.7 635 10.3 
All States 62,056 87.2 9,143 12.8 


Note. Percentages are based on the number of valid responses. Missing = 253. 


Table 3. Student Survey Math PBT (EOY): Q1. Did you understand all of the directions read by the person who 


gave you the test? 


Yes No 
State n % n % 
Arizona 7,021 74.2 2,437 25.8 
Arkansas 3,103 73.1 1,141 26.9 
Colorado 1,443 74.2 502 25.8 
District of Columbia 76 87.4 11 12.6 
Illinois 7,662 80.7 1,834 19.3 
Louisiana 371 78.9 99 21.1 
Maryland 6,015 79.9 1,510 20.1 
Massachusetts 5,359 76.7 1,629 23.3 
Mississippi 5,634 82.1 1,232 17.9 
New Jersey 4 66.7 2 33.3 
New York 1,271 78.4 351 21.6 
Ohio 6,082 81.5 1,380 18.5 
Rhode Island 1,192 84.8 214 15.2 
Tennessee 1,778 76.6 544 23.4 
All States 47,011 78.5 12,886 21.5 
Note. Percentages are based on the number of valid responses. Missing = 176. 
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Table 4. Student Survey Math PBT (PBA): Q2. How often was it hard to understand the directions for the questions 
on this test? 


Almost Always Most of the Time Some of the Time Almost Never 
State n % n % n % n % 
Arizona 1,467 15.5 2,631 27.7 4,394 46.3 996 10.5 
Arkansas 593 11.8 1,150 22.9 2,467 49.2 806 16.1 
Colorado 180 11.6 395 25.4 763 49.0 218 14.0 
District of Columbia 8 4.7 39 23.1 103 60.9 19 11.2 
Illinois 1,708 14.2 3,211 26.8 5,663 47.2 1,418 11.8 
Louisiana 15 6.8 44 20.1 129 58.9 31 14.2 
Maryland 810 11.1 1,773 24.2 3,682 50.3 1,048 14.3 
Massachusetts 848 12.1 1,745 24.9 3,360 47.9 1,061 15.1 
Mississippi 1,216 16.0 1,924 25.3 3,524 46.3 954 12.5 
New York 233 14.3 412 25.3 754 46.3 228 14.0 
Ohio 1,484 13.5 2,480 22.6 5,329 48.6 1,664 15.2 
Rhode Island 328 16.0 486 23.8 936 45.7 296 14.5 
Tennessee 768 12.4 1,362 22.0 2,998 48.5 1,052 17.0 
All States 9,658 13.6 17,652 24.8 34,102 47.9 9,791 13.8 


Note. Percentages are based on the number of valid responses. Missing = 249. 


Table 5. Student Survey Math PBT (EOY): Q2. How often was it hard to understand the directions for the questions 
on this test? 


Almost Always Most of the Time Some of the Time Almost Never 
State n % n % n % n % 
Arizona 1,161 12.3 2,145 22.7 4,647 49.2 1,490 15.8 
Arkansas 481 11.3 916 21.6 2,015 47.4 835 19.7 
Colorado 192 9.9 398 20.5 972 50.0 382 19.7 
District of Columbia 14 16.1 23 26.4 31 35.6 19 21.8 
Illinois 1,183 12.5 2,108 22.2 4,575 48.2 1,626 17.1 
Louisiana 36 7.6 86 18.3 252 53.5 97 20.6 
Maryland 719 9.6 1,483 19.7 3,771 50.1 1,550 20.6 
Massachusetts 706 10.1 1,400 20.0 3,440 49.2 1,440 20.6 
Mississipp1 879 12.8 1,523 22.2 3,195 46.6 1,266 18.4 
New Jersey 0 0.0 2 33.3 2 33.3 2 33.3 
New York 161 10.0 334 20.6 795 49.1 328 20.3 
Ohio 816 10.9 1,488 20.0 3,693 49.5 1,457 19.5 
Rhode Island 163 11.6 273 19.4 708 50.4 261 18.6 
Tennessee 331 14.2 458 19.7 1,087 46.7 450 19.3 
All States 6,842 11.4 1,2637 21.1 29,183 48.7 11,203 18.7 


Note. Percentages are based on the number of valid responses. Missing = 208. 
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Table 6. Student Survey Math PBT (PBA): Q3. How many questions asked you about things you have not learned in 
school this year? 


All of Them Most of Them Few of Them None of Them 
State n % n % n % n % 
Arizona 410 4.3 2,226 23.5 5,040 53.1 1,814 19.1 
Arkansas 125 25 1,104 22.0 2,838 56.6 944 18.8 
Colorado 46 3.0 399 25.6 834 53.6 278 17.9 
District of Columbia 4 2.4 31 18.3 84 49.7 50 29.6 
Illinois 440 3.7 3,729 31.1 6,406 53.4 1,417 11.8 
Louisiana 5 2:3 42 19.2 125 57.1 47 21.5 
Maryland 271 3.7 1,742 23.8 4,104 56.1 1,194 16.3 
Massachusetts 251 3.6 1,890 26.9 3,854 55.0 1,018 14.5 
Mississippi 317 4.2 1,927 25.3 3,993 52.4 1,379 18.1 
New York 74 4.5 433 26.6 905 55.6 217 13.3 
Ohio 424 3.9 3,137 28.6 5,917 54.0 1,479 13.5 
Rhode Island 81 4.0 642 31.4 1,051 51.4 272 13.3 
Tennessee 210 3.4 1,294 21.0 3,359 54.4 1,311 21.2 
All States 2,658 3.7 18,596 26.1 38,510 54.1 11,420 16.0 


Note. Percentages are based on the number of valid responses. Missing = 268. 


Table 7. Student Survey Math PBT (EOY): Q3. How many questions asked you about things you have not learned 
in school this year? 


All of Them Most of Them Few of Them None of Them 
State n % n % n % n % 
Arizona 378 4.0 2,117 22.4 5,117 54.2 1,826 19.3 
Arkansas 145 3.4 976 23.0 2,334 54.9 794 18.7 
Colorado 44 2.3 426 21.9 1,101 56.7 370 19.1 
District of Columbia 4 4.6 32 36.8 40 46.0 11 12.6 
Illinois 348 3.7 3,032 32.0 4,952 52.2 1,151 12.1 
Louisiana 9 1.9 56 11.9 245 52.0 161 34.2 
Maryland 251 3.3 1,575 20.9 4,230 56.3 1,462 19.4 
Massachusetts 244 3.5 1,439 20.6 3,738 53.5 1,562 22.4 
Mississippi 315 4.6 1,558 22.7 3,382 49.3 1,607 23.4 
New Jersey 1 16.7 1 16.7 4 66.7 0 0.0 
New York 43 2.7 275 17.0 932 57.7 366 22.6 
Ohio 250 3.4 1,894 25.4 3,865 51.9 1,442 19.4 
Rhode Island 55 3.9 303 21.6 843 60.1 201 14.3 
Tennessee 84 3.6 505 21) 1,247 53.7 487 21.0 
All States 2,171 3.6 14,189 23.7 32,030 53.5 11,440 19.1 


Note. Percentages are based on the number of valid responses. Missing = 243. 
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Table 8. Student Survey Math PBT (PBA): Q4. How difficult was this test? 


Easier than School Work Same as School Work Harder than School Work 

State n % n % n % 

Arizona 341 3.6 2,846 30.1 6,283 66.3 
Arkansas 202 4.0 1,697 33.8 3,115 62.1 
Colorado 120 7.7 472 30.5 957 61.8 
District of Columbia 4 2.4 76 45.0 89 52.7 
Illinois 315 2.6 2,771 23.1 8,899 74.3 
Louisiana 19 8.7 63 28.8 137 62.6 
Maryland 339 4.6 2,338 32.0 4,618 63.3 
Massachusetts 277 4.0 2,117 30.3 4,601 65.8 
Mississippi 243 3.2 2,129 28.0 5,236 68.8 
New York 57 3.5 542 33.3 1,027 63.2 
Ohio 358 3.3 3,017 27.6 7,575 69.2 
Rhode Island 54 2.6 503 24.6 1,486 7247 
Tennessee 234 3.8 1,841 29.8 4,096 66.4 
All States 2,563 3.6 20,412 28.7 48,119 67.7 


Note. Percentages are based on the number of valid responses. Missing = 358. 


Table 9. Student Survey Math PBT (EOY): Q4. How difficult was this test? 
Easier than School Work Same as School Work Harder than School Work 


State n % n % n % 

Arizona 554 5.9 3,247 34.4 5,634 59.7 
Arkansas 218 5.1 1,417 33.3 2,614 61.5 
Colorado 116 6.0 745 38.4 1,077 55.6 
District of Columbia 6 6.9 22 25.3 59 67.8 
Illinois 348 3.7 2,466 26.0 6,657 70.3 
Louisiana 61 13.0 201 42.7 209 44.4 
Maryland 465 6.2 2,924 39.0 4,112 54.8 
Massachusetts 339 4.9 2,451 35.2 4,181 60.0 
Mississippi 313 4.6 2,219 32.4 4,317 63.0 
New Jersey 0 0.0 6 100.0 0 0.0 
New York 87 5.4 653 40.4 878 54.3 
Ohio 334 4.5 2,375 31.9 4,729 63.6 
Rhode Island 43 3.1 407 29.1 951 67.9 
Tennessee 83 3.6 655 28.2 1,585 68.2 
All States 2,967 5.0 19,788 33.1 37,003 61.9 


Note. Percentages are based on the number of valid responses. Missing = 315. 
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Table 10. Student Survey Math PBT (PBA): Q5. Did you have enough time to finish this test? 


Finished Very Early _ Finished On Time Rushed to Finish Did Not Finish 
State n % n % n % n % 
Arizona 1,988 21.0 5,425 57.3 984 10.4 1,063 11.2 
Arkansas 1,384 27.6 2,742 54.7 531 10.6 352 7.0 
Colorado 526 33.9 843 54.3 110 7.1 74 4.8 
District of Columbia 32 18.9 113 66.9 8 4.7 16 9.5 
Illinois 3,348 27.9 6,952 58.0 1,006 8.4 677 5.6 
Louisiana 52 23.7 127 58.0 10 4.6 30 13.7 
Maryland 2,093 28.6 3,925 53.7 705 9.6 587 8.0 
Massachusetts 1,593 22.8 3,829 54.8 764 10.9 805 11.5 
Mississipp1 1,893 24.9 4,629 60.8 450 5.9 638 8.4 
New York 468 28.9 706 43.6 139 8.6 306 18.9 
Ohio 3,494 32.0 5,834 53.4 785 7.2 811 74 
Rhode Island 608 29.7 1,157 56.6 129 6.3 151 7.4 
Tennessee 1,751 28.4 3,417 55.3 462 75 545 8.8 
All States 19,230 27.1 39,699 55.9 6,083 8.6 6,055 8.5 


Note. Percentages are based on the number of valid responses. Missing = 385. 


Table 11. Student Survey Math PBT (EOY): Q5. Did you have enough time to finish this test? 
Finished Very Early _ Finished On Time Rushed to Finish Did Not Finish 


State n % n % n % n % 

Arizona 2,207 23.4 5,652 59.9 976 10.3 601 6.4 
Arkansas 1,645 38.7 2,126 50.1 335 7.9 140 3.3 
Colorado 666 34.3 1,031 53.1 169 8.7 75 3.9 
District of Columbia 17 19.3 50 56.8 13 14.8 8 9.1 
Illinois 3,939 41.5 4,890 51.6 467 4.9 189 2.0 
Louisiana 111 23.6 298 63.3 30 6.4 32 6.8 
Maryland 2,698 35.9 3,990 53.1 519 6.9 312 4.1 
Massachusetts 2,039 29.2 4,008 575 613 8.8 311 4.5 
Mississippi 1,791 26.1 4,233 61.7 435 6.3 397 5.8 
New Jersey 6 100.0 0 0.0 0 0.0 0 0.0 
New York 532 33.1 778 48.4 131 8.1 167 10.4 
Ohio 2,547 34.2 4,209 56.6 471 6.3 215 2.9 
Rhode Island 479 34.2 800 57.1 62 4.4 60 4.3 
Tennessee 721 31.0 1,275 54.8 216 9.3 113 49 
All States 19,398 32.4 33,340 55.8 4,437 74 2,620 4.4 


Note. Percentages are based on the number of valid responses. Missing = 278. 
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Table 12. Student Survey Math PBT (PBA): Q6. How often do you use a computer or tablet at home? 
A Few Times a A Couple Times A Do Not Have at 


Every Day Week Month or Less Home 

State n % n % n % n % 

Arizona 4,191 44.2 3,096 32.7 1,227 12.9 964 10.2 
Arkansas 2,491 49.8 1,416 28.3 583 11.6 516 10.3 
Colorado TAT 48.1 516 33.2 190 12.2 100 6.4 
District of Columbia 70 41.9 68 40.7 20 12.0 9 5.4 
Illinois 6,558 54.7 3,590 29.9 1,162 9.7 686 Bes 
Louisiana 78 35.6 71 32.4 33 15.1 37 16.9 
Maryland 3,598 49.2 2,459 33.6 811 11.1 440 6.0 
Massachusetts 3,864 55.2 2,241 32.0 599 8.5 302 4.3 
Mississippi 3,393 44.6 2,194 28.8 1,018 13.4 1,006 13.2 
New York 787 48.3 554 34.0 188 11.5 99 6.1 
Ohio 6,158 56.3 3,150 28.8 1,012 9.3 619 Si] 
Rhode Island 1,126 55.1 634 31.0 189 9.3 94 4.6 
Tennessee 3,216 52.1 1,647 26.7 733 11.9 581 9.4 
All States 36,277 51.0 21,636 30.4 7,765 10.9 5,453 veel 


Note. Percentages are based on the number of valid responses. Missing = 321. 


Table 13. Student Survey Math PBT (EOY): Q6. How often do you use a computer or tablet at home? 
A Few Times a A Couple Times A Do Not Have at 


Every Day Week Month or Less Home 

State n % n % n % n % 

Arizona 4,349 46.1 2,989 31.7 1,262 13.4 833 8.8 
Arkansas 2,131 50.2 1,170 27.6 502 11.8 442 10.4 
Colorado 953 49.3 614 31.7 237 12.3 130 6.7 
District of Columbia 59 67.0 20 22.7 7 8.0 2 2.3 
Illinois 5,239 55.3 2,654 28.0 986 10.4 597 6.3 
Louisiana 190 40.4 167 35.5 61 13.0 52 11.1 
Maryland 3,822 50.8 2,417 32.2 899 12.0 379 5.0 
Massachusetts 3,918 56.1 2,188 31.3 632 9.0 247 3.5 
Mississippi 3,089 45.1 1,968 28.7 1,011 14.8 782 11.4 
New Jersey 3 50.0 2 33.3 0 0.0 1 16.7 
New York 780 48.4 548 34.0 205 12.7 80 5.0 
Ohio 3,948 53.1 2,115 28.4 785 10.6 590 7.9 
Rhode Island 725 51.8 467 33.4 140 10.0 68 4.9 
Tennessee 1,172 50.5 605 26.1 296 12.8 248 10.7 
All States 30,378 50.8 17,924 30.0 7,023 11.7 4,451 7.4 


Note. Percentages are based on the number of valid responses. Missing = 297. 
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Table 14. Student Survey Math PBT (PBA): Q7. How often do you use a computer or tablet in school? 
A Few Times a A Couple Times A Do Not Use in 


Every Day Week Month or Less School 

State n % n % n % n % 

Arizona 1,338 14.1 3,503 37.0 3,533 37.3 1,104 11.6 
Arkansas 911 18.2 2,080 41.5 1,727 34.5 293 5.8 
Colorado 399 25.6 700 45.0 388 24.9 69 44 
District of Columbia 16 9.5 102 60.4 37 21.9 14 8.3 
Illinois 1,739 14.5 4,856 40.5 4,869 40.6 527 4.4 
Louisiana 53 24.2 100 45.7 4] 18.7 25 11.4 
Maryland 681 9.3 2,728 37.3 3,191 43.7 708 9.7 
Massachusetts 654 9.3 2,594 37.0 3,144 44.9 611 8.7 
Mississippi 1,962 25.8 2,375 31.2 2,426 31.9 844 11.1 
New York 153 9.4 487 29.9 710 43.5 281 17.2 
Ohio 1,654 15.1 4,660 42.6 4,043 37.0 581 5.3 
Rhode Island 159 7.8 820 40.1 943 46.1 123 6.0 
Tennessee 862 14.0 2,313 37.5 2,/22 44.1 270 4.4 
All States 10,581 14.9 27,318 38.4 27,774 39.1 5,450 7.7 


Note. Percentages are based on the number of valid responses. Missing = 329. 


Table 15. Student Survey Math PBT (EOY): Q7. How often do you use a computer or tablet in school? 
A Few Times a A Couple Times A Do Not Use in 


Every Day Week Month or Less School 

State n % n % n % n % 

Arizona 1,315 14.0 3,379 35.9 3,641 38.6 1,088 11.5 
Arkansas 1,139 26.8 1,888 44.5 1,049 24.7 167 3.9 
Colorado 524 27.1 792 41.0 516 26.7 101 5.2 
District of Columbia 11 12.6 43 49.4 28 32.2 B} a7 
Illinois 2,001 21.1 4,277 45.1 2,843 30.0 358 3.8 
Louisiana 79 16.8 238 50.5 96 20.4 58 12.3 
Maryland 733 9.8 2,802 37.3 3,424 45.6 555 7.4 
Massachusetts 686 9.8 2,615 37:5 3,192 45.7 485 7.0 
Mississippi 1,826 26.6 2,322 33.9 2,185 31.9 524 7.6 
New Jersey 0 0.0 4 66.7 1 16.7 1 16.7 
New York 219 13.6 626 38.9 540 33.5 225 14.0 
Ohio 1,332 17.9 3,355 45.1 2,372 31.9 380 5.1 
Rhode Island 138 9.9 660 47.1 497 35.5 106 7.6 
Tennessee 280 12.1 1,033 44.5 913 39.3 95 4.1 
All States 10,283 17.2 24,034 40.2 21,297 35.6 4,148 6.9 


Note. Percentages are based on the number of valid responses. Missing = 311. 
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Table 16. Student Survey Math PBT (PBA): Q8. When writing a story or essay, how often do you use a computer or 
tablet? 


All of the Time Most of the Time Some of the Time Never 

State n % n % n % n % 

Arizona 1,026 10.8 2,194 23.2 3,681 38.9 2,573 27.2 
Arkansas 425 8.5 1,046 20.9 2,005 40.0 1,534 30.6 
Colorado 144 9.3 485 31.2 640 41.2 284 18.3 
District of Columbia 24 14.2 34 20.1 75 44.4 36 21.3 
Illinois 1,966 16.4 3,536 29.5 4,436 37.0 2,046 17.1 
Louisiana 14 6.4 22 10.1 48 22.0 134 61.5 
Maryland 816 11.2 1,779 24.4 2,958 40.5 1,750 24.0 
Massachusetts 1,167 16.7 1,844 26.3 2,407 34.3 1,591 22.7 
Mississippi 1,020 13.4 1,777 23.4 2,863 37.6 1,950 25.6 
New York 209 12.9 352 21.7 560 34.5 504 31.0 
Ohio 1,622 14.8 2,988 27.3 4,295 39.2 2,050 18.7 
Rhode Island 356 17.4 632 30.9 794 38.8 266 13.0 
Tennessee 634 10.3 1,509 24.4 2,703 43.8 1,329 215 
All States 9,423 13.2 18,198 25.6 27,465 38.6 16,047 22.6 


Note. Percentages are based on the number of valid responses. Missing = 319. 


Table 17. Student Survey Math PBT (EOY): Q8. When writing a story or essay, how often do you use a computer or 
tablet? 


All of the Time Most of the Time Some of the Time Never 

State n % n % n % n % 

Arizona 1,007 10.7 2,274 24.1 3,624 38.4 2,528 26.8 
Arkansas 375 8.8 846 19.9 1,733 40.8 1,289 30.4 
Colorado 234 12.1 551 28.5 809 41.8 340 17.6 
District of Columbia 24 27.6 27 31.0 22 25.3 14 16.1 
Illinois 1,777 18.7 3,099 32.7 3,194 33.7 1,417 14.9 
Louisiana 37 7.9 54 11.5 160 34.0 220 46.7 
Maryland 906 12.1 1,836 24.5 3,076 41.0 1,689 22.5 
Massachusetts 1,146 16.4 2,099 30.0 2,470 35.3 1,273 18.2 
Mississippi 933 13.6 1,642 24.0 2,556 37.3 1,723 25.1 
New Jersey 1 16.7 1 16.7 3 50.0 1 16.7 
New York 122 7.6 347 21.5 656 40.7 487 30.2 
Ohio 1,049 14.1 2,038 27.4 2,964 39.9 1,386 18.6 
Rhode Island 228 16.3 408 29.1 540 38.5 226 16.1 
Tennessee 291 12.5 624 26.9 981 42.2 427 18.4 
All States 8,130 13.6 15,846 26.5 22,788 38.1 13,020 21.8 


Note. Percentages are based on the number of valid responses. Missing = 289. 
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Table 18. Student Survey Math PBT (PBA): Q9. Would you rather take this test on paper OR on a computer or 


tablet? 

On a Computer or Tablet 
State n % n % 
Arizona 3,217 34.1 6,230 65.9 
Arkansas 1,697 33.9 3,305 66.1 
Colorado 491 31.8 1,055 68.2 
District of Columbia 22 13.0 147 87.0 
Illinois 4,485 37.5 7,478 62.5 
Louisiana 85 38.8 134 61.2 
Maryland 1,952 26.7 5,350 73.3 
Massachusetts 2,772 39.6 4,222 60.4 
Mississipp1 2,095 27.6 5,498 72.4 
New York 646 39.9 973 60.1 
Ohio 4,632 42.5 6,275 57.5 
Rhode Island 764 37.4 1,278 62.6 
Tennessee 2,137 34.7 4,016 65.3 
All States 24,995 35.2 45,961 64.8 


Note. Percentages are based on the number of valid responses. Missing = 496. 


Table 19. Student Survey Math PBT (EOY): Q9. Would you rather take this test on paper OR on a computer or 


tablet? 

On a Computer or Tablet 
State n % n % 
Arizona 3,263 34.7 6,151 65.3 
Arkansas 1,491 35.2 2,744 64.8 
Colorado 684 35.4 1,246 64.6 
District of Columbia 20 23.0 67 77.0 
Illinois 3,697 39.1 5,754 60.9 
Louisiana 182 38.6 289 61.4 
Maryland 1,929 25.7 5,572 74.3 
Massachusetts 2,790 40.1 4,166 59.9 
Mississippi 2,032 29.7 4,801 70.3 
New Jersey 2 33.3 4 66.7 
New York 638 39.7 968 60.3 
Ohio 3,043 41.0 4,377 59.0 
Rhode Island 540 38.6 859 61.4 
Tennessee 842 36.3 1,476 63.7 
All States 21,153 35.5 38,474 64.5 
Note. Percentages are based on the number of valid responses. Missing = 446. 
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Appendix M — Item-level Results from the Student Survey for 
English Language Arts: Computer/Tablet-based Administration 


Table 1. Student Survey Responses by State (ELA CBT) 


Performance-Based Assessment 


End-of-Y ear Assessment 


State n % of Total Responses n % of Total Responses 
Arizona 18,867 8.5 13,672 8.5 
Arkansas 10,975 5.0 8,371 5.2 
Colorado 5,089 2.3 5,372 3.3 
District of Columbia 663 0.3 2,064 1.3 
Florida 2,446 1.1 2,098 1.3 
Illinois 29,770 13.5 23,193 14.4 
Louisiana 11,753 5.3 7,042 44 
Maryland 17,372 7.9 17,035 10.6 
Massachusetts 21,348 9.7 18,609 11.6 
Mississippi 14,185 6.4 9,933 6.2 
New Jersey 29,198 13.2 21,219 13.2 
New Mexico 3,582 1.6 1,800 1.1 
New York 2,714 1.2 3,583 2.2 
Ohio 30,943 14.0 20,417 12.7 
Rhode Island 5,252 2.4 2,608 1.6 
Tennessee 17,001 7.7 3,804 2.4 
All States 221,158 100.0 160,820 100.0 
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Table 2. Student Survey ELA CBT (PBA): Q1. Did you understand all of the directions read by the person who gave 
you the test? 


Yes No 

State n % n % 

Arizona 17,309 91.8 1,541 8.2 
Arkansas 10,375 94.6 595 5.4 
Colorado 4,691 92.3 393 77 
District of Columbia 633 95.6 29 4.4 
Florida 2,303 94.2 143 5.8 
Illinois 27,821 93.5 1,932 6.5 
Louisiana 11,118 94.7 624 5.3 
Maryland 16,401 94.5 960 3.5 
Massachusetts 20,263 95.0 1,069 5.0 
Mississippi 13,454 95.0 715 5.0 
New Jersey 27,536 94.4 1,647 5.6 
New Mexico 3,332 93.1 247 6.9 
New York 2,526 93.2 185 6.8 
Ohio 28,768 93.1 2,148 6.9 
Rhode Island 4,960 94.5 289 5:5 
Tennessee 16,058 94.6 923 5.4 
All States 207,548 93.9 13,440 6.1 


Note. Percentages are based on the number of valid responses. Missing = 170. 


Table 3. Student Survey ELA CBT (EOY): Q1. Did you understand all of the directions read by the person who 
gave you the test? 


Yes No 

State n % n % 

Arizona 11,566 84.6 2,102 15.4 
Arkansas 7,323 87.5 1,043 12.5 
Colorado 4,585 85.4 786 14.6 
District of Columbia 1,815 88.0 248 12.0 
Florida 1,870 89.2 227 10.8 
Illinois 19,899 85.8 3,281 14.2 
Louisiana 6,145 87.3 895 12.7 
Maryland 14,646 86.1 2,368 13.9 
Massachusetts 16,019 86.1 2,582 13.9 
Mississippi 8,644 87.1 1,279 12.9 
New Jersey 18,416 86.8 2,797 13.2 
New Mexico 1,590 88.4 208 11.6 
New York 3,050 85.1 533 14.9 
Ohio 16,942 83.0 3,464 17.0 
Rhode Island 2,235 85.7 372 14.3 
Tennessee 3,263 85.9 537 14.1 
All States 138,008 85.9 22,722 14.1 


Note. Percentages are based on the number of valid responses. Missing = 90. 
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Table 4. Student Survey ELA CBT (PBA): Q2. How often was it hard to understand the directions for the questions 
on this test? 


Almost Always Most of the Time Some of the Time Almost Never 
State n % n % n % n % 
Arizona 1,199 6.4 3,187 16.9 9,944 52.8 4,497 23.9 
Arkansas 540 49 1,453 13.3 5,229 47.7 3,738 34.1 
Colorado 290 5.7 708 13.9 2,609 51.4 1,471 29.0 
District of Columbia 42 6.3 87 13.1 316 47.7 217 32.8 
Florida 137 5.6 365 15.0 1,224 50.2 714 29.3 
Illinois 1,572 5.3 4,140 13.9 15,578 52.4 8,441 28.4 
Louisiana 741 6.3 1,731 14.8 5,892 50.2 3,369 28.7 
Maryland 1,050 6.1 2,504 14.4 8,496 49.0 5,289 30.5 
Massachusetts 1,094 5.1 2,801 13.1 10,717 50.3 6,691 31.4 
Mississippi 815 5.8 2,182 15.4 7,401 52.3 3,766 26.6 
New Jersey 1,461 5.0 3,920 13.4 14,849 50.9 8,939 30.6 
New Mexico 220 6.2 604 16.9 1,705 47.7 1,043 29.2 
New York 185 6.8 424 15.7 1,375 50.8 722 26.7 
Ohio 1,950 6.3 4,233 13.7 16,024 51.9 8,678 28.1 
Rhode Island 322 6.1 738 14.1 2,518 48.0 1,666 31.8 
Tennessee 1,023 6.0 2,292 13.5 8,356 49.3 5,291 31.2 
All States 12,641 5.7 31,369 14.2 112,233 50.8 64,532 29.2 


Note. Percentages are based on the number of valid responses. Missing = 383. 


Table 5. Student Survey ELA CBT (EOY): Q2. How often was it hard to understand the directions for the questions 
on this test? 


Almost Always Most of the Time Some of the Time Almost Never 
State n % n % n % n % 
Arizona 992 7.3 2,410 17.7 7,020 51.4 3,224 23.6 
Arkansas 539 6.4 1,113 13.3 4,170 49.9 2,541 30.4 
Colorado 292 5.4 773 14.4 2,821 52.6 1,480 27.6 
District of Columbia 175 8.5 378 18.4 949 46.2 551 26.8 
Florida 144 6.9 342 16.3 986 47.1 622 29.7 
Illinois 1,405 6.1 3,265 14.1 12,026 51.9 6,473 27.9 
Louisiana 547 7.8 1,176 16.7 3,360 47.8 1,945 27.7 
Maryland 1,187 7.0 2,639 15.5 8,231 48.4 4,952 29.1 
Massachusetts 1,280 6.9 2,509 13.5 8,935 48.1 5,846 31.5 
Mississipp1 797 8.0 1,705 17.2 4,833 48.8 2,576 26.0 
New Jersey 1,229 5.8 3,103 14.7 10,640 50.2 6,208 29.3 
New Mexico 124 6.9 326 18.1 900 50.1 448 24.9 
New York 239 6.7 569 15.9 1,825 51.0 946 26.4 
Ohio 1,684 8.3 3,203 15.7 10,235 50.3 5,244 25.7 
Rhode Island 178 6.8 406 15.6 1,280 49.2 740 28.4 
Tennessee 276 7.3 625 16.5 1,855 48.9 1,036 27.3 
All States 11,088 6.9 24,542 15.3 80,066 49.9 44,832 27.9 


Note. Percentages are based on the number of valid responses. Missing = 292. 
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Table 6. Student Survey ELA CBT (PBA): Q3. How many questions asked you about things you have not learned in 
school this year? 


All of Them Most of Them Few of Them None of Them 
State n % n % n % n % 
Arizona 673 3.6 2,183 11.6 9,218 49.0 6,729 35.8 
Arkansas 393 3.6 981 9.0 5,301 48.4 4,275 39.0 
Colorado 169 3.3 491 9.7 2,524 49.7 1,890 37.2 
District of Columbia 24 3.6 65 9.8 293 44.3 279 42.2 
Florida 58 2.4 243 10.0 1,253 51.4 886 36.3 
Illinois 1,070 3.6 3,631 12.2 15,948 53.7 9,063 30.5 
Louisiana 455 3.9 1,354 11.6 5,825 49.7 4,086 34.9 
Maryland 796 4.6 2,107 12.2 8,705 50.3 5,705 33.0 
Massachusetts 671 3.2 1,997 9.4 10,198 47.9 8,415 39.5 
Mississippi 533 3.8 1,534 10.8 6,459 45.6 5,634 39.8 
New Jersey 1,108 3.8 3,673 12.6 14,345 49.2 10,026 34.4 
New Mexico 121 3.4 381 10.7 1,814 50.8 1,255 35.1 
New York 151 5.6 336 12.4 1,271 47.0 944 34.9 
Ohio 1,226 4.0 3,839 12.4 16,461 53.3 9,343 30.3 
Rhode Island 218 4.2 639 12.2 2,603 49.7 1,777 33.9 
Tennessee 600 3.5 1,669 9.8 8,314 49.0 6,368 37.6 
All States 8,266 3.7 25,123 11.4 110,532 50.1 76,675 34.8 


Note. Percentages are based on the number of valid responses. Missing = 562. 


Table 7. Student Survey ELA CBT (EOY): Q3. How many questions asked you about things you have not learned in 
school this year? 


All of Them Most of Them Few of Them None of Them 
State n % n % n % n % 
Arizona 604 4.4 1,884 13.8 7,205 52.8 3,946 28.9 
Arkansas 349 4.2 1,037 12.4 4,297 51.4 2,675 32.0 
Colorado 161 3.0 727 13.5 2,952 55.0 1,529 28.5 
District of Columbia 128 6.2 279 13.6 941 45.9 702 34.2 
Florida 70 3.3 212 10.1 961 45.9 852 40.7 
Illinois 887 3.8 3,210 13.9 12,809 55.3 6,257 27.0 
Louisiana 395 5.6 950 13.5 3,489 49.7 2,189 31.2 
Maryland 826 4.9 2,165 12.7 8,623 50.7 5,387 31.7 
Massachusetts 821 4.4 2,143 11.5 8,833 47.6 6,766 36.4 
Mississippi 494 5.0 1,208 12.2 4,610 46.5 3,610 36.4 
New Jersey 940 4.4 2,740 12.9 10,646 50.3 6,843 32.3 
New Mexico 86 4.8 258 14.4 1,003 55.8 450 25.0 
New York 191 5.3 456 12.8 1,816 50.8 1,109 31.0 
Ohio 1,096 5.4 3,042 14.9 10,759 52.9 5,459 26.8 
Rhode Island 131 5.0 389 15.0 1,353 52.0 728 28.0 
Tennessee 158 4.2 502 13.2 1,870 49.3 1,261 33.3 
All States 7,337 4.6 21,202 13.2 82,167 51.2 49,763 31.0 


Note. Percentages are based on the number of valid responses. Missing = 351. 
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Table 8, Student Survey ELA CBT (PBA): Q4. How difficult was this test? 


Easier than School Work Same as School Work Harder than School Work 
State n % n % n % 
Arizona 2,973 15.8 10,188 54.2 5,633 30.0 
Arkansas 1,818 16.6 6,107 55.8 3,026 27.6 
Colorado 956 18.8 2,933 57.8 1,184 23.3 
District of Columbia 136 20.5 407 61.5 119 18.0 
Florida 305 12.5 1,410 57.8 724 29.7 
Illinois 4,340 14.6 15,468 52.1 9,887 33.3 
Louisiana 1,709 14.6 6,463 55.1 3,551 30.3 
Maryland 3,111 18.0 9,659 55.8 4,551 26.3 
Massachusetts 3,514 16.5 11,874 55.8 5,897 27.7 
Mississippi 1,918 13.6 7,585 53.6 4,646 32.8 
New Jersey 3,629 12.5 15,369 52.7 10,149 34.8 
New Mexico 635 17.8 1,984 55.5 957 26.8 
New York 385 14.3 1,543 S76 772 28.6 
Ohio 4,070 13.2 15,514 50.3 11,270 36.5 
Rhode Island 613 11.7 2,629 50.2 1,993 38.1 
Tennessee 2,359 13.9 9,027 53.3 5,549 32.8 
All States 32,471 14.7 118,160 53.6 69,908 31.7 
Note. Percentages are based on the number of valid responses. Missing = 619. 
Table 9. Student Survey ELA CBT (EOY): Q4. How difficult was this test? 

Easier than School Work Same as School Work Harder than School Work 
State n % n % n % 
Arizona 1,818 13.3 7,339 53.8 4,482 32.9 
Arkansas 1,210 14.5 4,612 S22 2,539 30.4 
Colorado 844 15.7 3,130 58.3 1,392 25.9 
District of Columbia 331 16.1 1,236 60.2 487 23.7 
Florida 244 11.7 1,219 58.3 629 30.1 
Illinois 2,850 12.3 12,128 52.4 8,182 35.3 
Louisiana 986 14.0 3,796 54.1 2,240 31.9 
Maryland 2,566 15.1 9,584 56.4 4,853 28.5 
Massachusetts 2,635 14.2 9,832 52.9 6,102 32.9 
Mississippi 1,048 10.6 5,222 52.7 3,641 36.7 
New Jersey 2,291 10.8 11,238 53.1 7,641 36.1 
New Mexico 217 12.1 1,055 58.7 524 29.2 
New York 435 12.2 2,052 57.4 1,090 30.5 
Ohio 2,235 11.0 9,927 48.8 8,190 40.2 
Rhode Island 223 8.6 1,210 46.5 1,167 44.9 
Tennessee 421 11.1 1,961 51.7 1,410 37.2 
All States 20,354 12.7 85,541 53.3 54,569 34.0 


Note. Percentages are based on the number of valid responses. Missing = 356. 
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Table 10. Student Survey ELA CBT (PBA): Q5. Did you have enough time to finish this test? 


Finished Very Early Finished On Time Rushed to Finish Did Not Finish 

State n % n % n % n % 

Arizona 6,001 31.9 11,224 59.7 1,171 6.2 397 2.1 
Arkansas 5,223 47.7 5,201 47.5 391 3.6 141 1.3 
Colorado 2,085 41.1 2,652 52.3 267 5.3 70 1.4 
District of Columbia 196 29.6 409 61.8 35 5.3 22 3.3 
Florida 738 30.2 1,478 60.6 170 7.0 54 2.2 
Illinois 12,739 42.9 15,339 51.6 1,234 4.2 387 1.3 
Louisiana 3,959 33.8 6,851 58.4 660 5.6 257 2.2, 
Maryland 8,476 48.9 7,892 45.6 666 3.8 287 1.7 
Massachusetts 8,213 38.6 11,611 54.6 1,049 4.9 411 1.9 
Mississipp1 4,698 33.2 8,757 61.9 482 3.4 217 1.5 
New Jersey 12,334 42.3 14,597 50.1 1,712 5.9 501 1.7 
New Mexico 1,087 30.4 2,222 62.2 171 4.8 93 2.6 
New York 946 35.0 1,415 52.4 242 9.0 97 3.6 
Ohio 13,326 43.2 15,509 50.3 1,627 5.3 389 1.3 
Rhode Island 2,053 39.1 2,901 55:3 211 4.0 79 1.5 
Tennessee 6,820 40.3 9,046 53.4 778 4.6 288 1.7 
All States 88,894 40.3 117,104 53.1 10,866 4.9 3,690 1.7 


Note. Percentages are based on the number of valid responses. Missing = 604. 


Table 11. Student Survey ELA CBT (EOY): Q5. Did you have enough time to finish this test? 


Finished Very Early Finished On Time Rushed to Finish Did Not Finish 
State n % n % n % n % 
Arizona 4,529 33.2 7,824 57.3 1,016 74 275 2.0 
Arkansas 4,229 50.6 3,735 44.7 323 3.9 77 0.9 
Colorado 2,417 45.1 2,642 49.2 253 4.7 53 1.0 
District of Columbia 630 30.7 1,226 59.8 136 6.6 58 2.8 
Florida 703 33.6 1,210 57.9 140 6.7 38 1.8 
Illinois 10,211 44.1 11,736 50.7 997 43 217 0.9 
Louisiana 2,293 32.6 4,208 59.9 408 5.8 117 1.7 
Maryland 8,500 50.0 7,610 44.8 702 4.1 190 1.1 
Massachusetts 7,521 40.5 9,858 53.1 967 D2 217 1.2 
Mississippi 3,391 34.2 6,006 60.6 413 4.2 106 1.1 
New Jersey 9,844 46.5 9,824 46.4 1,201 57 298 1.4 
New Mexico 515 28.7 1,131 62.9 131 7.3 20 1.1 
New York 1,392 38.9 1,815 50.8 304 8.5 64 1.8 
Ohio LAs 38.0 10,681 52.5 1,634 8.0 292 1.4 
Rhode Island 1,008 38.7 1,468 56.4 99 3.8 29 1.1 
Tennessee 1,501 39.6 2,012 53.1 240 6.3 39 1.0 
All States 66,421 41.4 82,986 51.7 8,964 5.6 2,090 1.3 


Note. Percentages are based on the number of valid responses. Missing = 359. 
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Table 12. Student Survey ELA CBT (PBA): Q6. How often do you use a computer or tablet at home? 


A Couple Times A Do Not Have at 
Every Day A Few Times a Week Month or Less Home 

State n % n % n % n % 
Arizona 9,012 48.0 5,709 30.4 2,259 12.0 1,809 9.6 
Arkansas 5,313 48.5 3,073 28.1 1,343 12.3 1,221 11.2 
Colorado 2,296 45.3 1,713 33.8 674 13.3 384 7.6 
District of Columbia 295 44.7 222 33.6 91 13.8 52 7.9 
Florida 1,172 48.1 848 34.8 261 10.7 158 6.5 
Illinois 16,916 57.0 8,421 28.4 2,691 9.1 1,654 5.6 
Louisiana 5,849 49.9 3,303 28.2 1,497 12.8 1,065 9.1 
Maryland 9,085 52.5 5,291 30.6 1,855 10.7 1,080 6.2 
Massachusetts 12,288 57.8 6,406 30.1 1,780 8.4 793 3.7 
Mississipp1 6,693 47.3 4,021 28.4 1,800 12.7 1,636 11.6 
New Jersey 17,872 61.3 7,972 27.4 2,284 7.8 1,008 3:5 
New Mexico 1,620 45.3 1,125 31.5 438 12.3 390 10.9 
New York 1,367 50.6 798 29.5 323 12.0 213 7.9 
Ohio 17,984 58.3 8,087 26.2 2,880 9.3 1,882 6.1 
Rhode Island 2,926 55.8 1,674 31.9 415 7.9 225 4.3 
Tennessee 8,631 51.0 4,534 26.8 2,097 12.4 1,648 Od 
All States 119,319 54.1 63,197 28.7 22,688 10.3 15,218 6.9 


Note. Percentages are based on the number of valid responses. Missing = 736. 


Table 13. Student Survey ELA CBT (EOY): Q6. How often do you use a computer or tablet at home? 


A Couple Times A Do Not Have at 

Every Day A Few Times a Week Month or Less Home 
State n % n % n % n % 
Arizona 6,533 47.9 4,140 30.4 1,761 12.9 1,204 8.8 
Arkansas 4,032 48.2 2,387 28.6 1,083 13.0 858 10.3 
Colorado 2,600 48.5 1,720 32.1 686 12.8 358 6.7 
District of Columbia 919 44.9 735 35.9 226 11.0 166 8.1 
Florida 933 44.7 773 37.0 261 12.5 122 5.8 
Illinois 12,610 54.5 6,897 29.8 2,390 10.3 1,253 5.4 
Louisiana 3,442 49.0 1,962 27.9 1,005 14.3 613 8.7 
Maryland 8,645 50.9 5,357 31.5 2,064 12.1 932 5.5 
Massachusetts 10,410 56.1 5,721 30.8 1,752 9.4 675 3.6 
Mississippi 4,394 44.3 2,966 29.9 1,498 15.1 1,053 10.6 
New Jersey 12,630 59.7 5,896 27.9 1,875 8.9 753 3.6 
New Mexico 799 44.4 552. 30.7 265 14.7 182 10.1 
New York 1,748 48.9 1,178 33.0 437 12.2 209 5.9 
Ohio 11,196 55.0 5,488 27.0 2,260 11.1 1,397 6.9 
Rhode Island 1,415 54.4 862 33.1 228 8.8 96 3.7 
Tennessee 1,892 50.0 970 25.6 544 14.4 378 10.0 
All States 84,198 52.5 47,604 29.7 18,335 11.4 10,249 6.4 
Note. Percentages are based on the number of valid responses. Missing = 434. 
Updated February 17, 2015 Page M-7 


PARCC Quality of Test Administration 


Table 14. Student Survey ELA CBT (PBA): Q7. How often do you use a computer or tablet in school? 


A Couple Times A Do Not Use 
Every Day A Few Times a Week Month or Less in School 

State n % n % n % n % 

Arizona 3,538 18.8 7,176 38.2 6,504 34.6 1,555 8.3 
Arkansas 3,035 27.7 4,681 42.8 2,871 26.2 355 32, 
Colorado 1,037 20.5 2,313 45.7 1,543 30.5 169 3.3 
District of Columbia 108 16.4 278 42.1 230 34.8 44 6.7 
Florida 375 15.4 893 36.7 964 39.6 203 8.3 
Illinois 7,252 24.4 13,793 46.5 7,834 26.4 789 2.7 
Louisiana 2,660 22.7 4,352 37,2 3,947 33.7 750 6.4 
Maryland 2,591 15.0 6,948 40.2 6,949 40.2 816 47 
Massachusetts 4,030 19.0 9,279 43.7 7,056 33.2 891 4.2 
Mississippi 4,377 30.9 5,159 36.5 3,790 26.8 817 5.8 
New Jersey 5,626 19.3 12,601 43.3 9,981 34.3 897 3.1 
New Mexico 668 18.7 1,387 38.8 1,302 36.5 214 6.0 
New York 403 14.9 1,139 42.1 961 35.6 200 7.4 
Ohio 7,446 24.2 13,661 44.3 8,903 28.9 810 2.6 
Rhode Island 661 12.6 2,532 48.4 1,853 35.4 189 3.6 
Tennessee 2,950 17.5 6,909 40.9 6,402 37.9 638 3.8 
All States 46,757 21.2 93,101 42.3 71,090 32.3 9,337 4.2 


Note. Percentages are based on the number of valid responses. Missing = 873. 


Table 15. Student Survey ELA CBT (EOY): Q7. How often do you use a computer or tablet in school? 


A Couple Times A Do Not Use 
Every Day A Few Times a Week Month or Less in School 
State n % n % n % n % 
Arizona 2,412 17.7 5,170 38.0 5,028 36.9 1,012 74 
Arkansas 2,280 27.3 3,693 44.2 2,146 25.7 235 2.8 
Colorado 1,226 22.9 2,201 41.0 1,759 32.8 177 3.3 
District of Columbia 415 20.3 945 46.3 608 29.8 75 3.7 
Florida 295 14.1 747 35.8 865 41.5 179 8.6 
Illinois 5,570 24.1 10,627 45.9 6,261 27.1 688 3.0 
Louisiana 1,625 23.1 2,558 36.4 2,407 34.3 432 6.2 
Maryland 2,276 13.4 6,901 40.6 7,078 41.7 726 4.3 
Massachusetts 3,731 20.1 7,656 41.3 6,342 34.2 814 4.4 
Mississippi 2,920 29.5 3,297 33.3 3,005 30.3 687 6.9 
New Jersey 4,387 20.8 9,202 43.5 6,899 32.6 650 3.1 
New Mexico 418 23.3 632 35,2 637 35.5 108 6.0 
New York 477 13.4 1,530 42.8 1,326 37.1 238 6.7 
Ohio 5,125 25.2 8,654 42.6 5,895 29.0 647 3.2 
Rhode Island 370 14.2 1,345 51.7 793 30.5 95 3.6 
Tennessee 570 15.1 1,471 38.9 1,565 41.3 180 48 
All States 34,097 21.3 66,629 41.6 52,614 32.8 6,943 4.3 


Note. Percentages are based on the number of valid responses. Missing = 537. 
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Table 16. Student Survey ELA CBT (PBA): Q8. When writing a story or essay, how often do you use a computer or 
tablet? 


All of the Time Most of the Time Some of the Time Never 

State n % n % n % n % 

Arizona 2,199 11.7 4,720 25.1 7,880 42.0 3,970 21.2 
Arkansas 875 8.0 2,206 20.2 5,224 47.7 2,640 24.1 
Colorado 563 11.1 1,308 25.8 2,341 46.2 854 16.9 
District of Columbia 62 9.4 120 18.2 316 47.8 163 24.7 
Florida 181 74 471 19.4 1,058 43.5 724 29.7 
Illinois 4,815 16.2 9,289 31.3 11,591 39.1 3,978 13.4 
Louisiana 1,193 10.2 2,193 18.7 4,895 41.8 3,433 29.3 
Maryland 2,419 14.0 4,470 25.8 7,459 43.1 2,948 17.0 
Massachusetts 3,351 15.8 5,901 27.8 8,298 39.0 3,709 17.4 
Mississippi 1,679 11.9 3,353 23.7 5,943 42.1 3,154 22.3 
New Jersey 4,013 13.8 8,944 30.7 12,057 41.4 4,087 14.0 
New Mexico 505 14.1 1,006 28.2 1,408 39.4 652 18.3 
New York 312 11.6 647 24.0 1,169 43.4 567 21.0 
Ohio 4,725 15.3 9,144 29.7 12,151 39.4 4,797 15.6 
Rhode Island 808 15.4 1,594 30.5 2,095 40.0 735 14.0 
Tennessee 1,838 10.9 4,279 25.3 7,799 46.2 2,980 17.6 
All States 29,538 13.4 59,645 27.1 91,684 41.6 39,391 17.9 


Note. Percentages are based on the number of valid responses. Missing = 900. 


Table 17. Student Survey ELA CBT (EOY): Q8. When writing a story or essay, how often do you use a computer or 
tablet? 


All of the Time Most of the Time Some of the Time Never 

State n % n % n % n % 

Arizona 1,717 12.6 3,470 25.5 5,524 40.5 2,914 21.4 
Arkansas 813 9.7 1,924 23.0 3,730 44.6 1,892 22.6 
Colorado 653 12.2 1,632 30.4 2,308 43.0 770 14.4 
District of Columbia 322 15.8 523 25.6 811 39.7 388 19.0 
Florida 178 8.5 371 17.8 917 43.9 622 29.8 
Illinois 3,891 16.8 7,029 30.4 8,822 38.1 3,399 14.7 
Louisiana 811 11.5 1,379 19.6 2,805 39.9 2,028 28.9 
Maryland 2,257 13.3 4,208 24.8 7,474 44.0 3,056 18.0 
Massachusetts 3,263 17.6 5,246 28.3 6,976 37.6 3,065 16.5 
Mississippi 1,321 13.3 2,341 23.6 3,958 39.9 2,292 23.1 
New Jersey 3,100 14.7 6,345 30.0 8,489 40.1 3,217 15.2 
New Mexico 184 10.2 489 27.2 802 44.6 322 17.9 
New York 340 9.5 742 20.8 1,604 45.0 879 24.7 
Ohio 3,427 16.9 5,908 29.1 7,760 38.2 3,238 15.9 
Rhode Island 446 17.1 809 31.1 961 36.9 387 14.9 
Tennessee 457 12.1 918 24.2 1,661 43.8 753 19.9 
All States 23,180 14.5 43,334 27.0 64,602 40.3 29,222 18.2 


Note. Percentages are based on the number of valid responses. Missing = 482. 
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Table 18. Student Survey ELA CBT (PBA): Q9. Would you rather take this test on paper OR on a computer or 


tablet? 
On Paper On a Computer or Tablet 

State n % n % 
Arizona 3,768 20.1 14,990 79.9 
Arkansas 2,939 26.8 8,009 73.2 
Colorado 1,292 25.5 3,776 74.5 
District of Columbia 72 10.9 590 89.1 
Florida 486 19.9 1,952 80.1 
Illinois 6,497 21.9 23,150 78.1 
Louisiana 2,599 22.2 9,110 77.8 
Maryland 2,366 13.7 14,942 86.3 
Massachusetts 5,548 26.1 15,699 73.9 
Mississippi 2,920 20.6 11,227 79.4 
New Jersey 8,131 27.9 20,988 72.1 
New Mexico 674 18.9 2,897 81.1 
New York 829 30.8 1,866 69.2 
Ohio 9,769 31.7 21,040 68.3 
Rhode Island 972 18.6 4,265 81.4 
Tennessee 3,074 18.2 13,809 81.8 
All States 51,936 23.6 168,310 76.4 


Note. Percentages are based on the number of valid responses. Missing = 912. 


Table 19. Student Survey ELA CBT (EOY): Q9. Would you rather take this test on paper OR on a computer or 


tablet? 

On Paper On a Computer or Tablet 
State n % n % 
Arizona 3,122 22.9 10,510 771 
Arkansas 2,488 29.7 5,876 70.3 
Colorado 1,378 25.7 3,983 74.3 
District of Columbia 405 19.9 1,635 80.1 
Florida 466 22.3 1,622 77.7 
Illinois 6,012 26.0 17,128 74.0 
Louisiana 1,817 25.9 5,205 74.1 
Maryland 2,631 15.5 14,370 84.5 
Massachusetts 6,332 34.1 12,215 65.9 
Mississipp1 2,736 27.6 7,176 72.4 
New Jersey 7,072 33.5 14,065 66.5 
New Mexico 380 21.2 1,414 78.8 
New York 1,150 32.2 2,421 67.8 
Ohio 7,931 39.0 12,390 61.0 
Rhode Island 602 23.1 2,002 76.9 
Tennessee 905 23.9 2,877 76.1 
All States 45,427 28.3 114,889 71.7 
Note. Percentages are based on the number of valid responses. Missing = 504. 
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Table 20. Student Survey ELA CBT (PBA): Q10. What did you use to take this test? 


Desktop Computer Laptop Computer Tablet with a Keyboard Tablet without Keyboard 
State n % n % n % n % 
Arizona 12,196 65.0 5,526 29.5 815 4.3 222 1.2 
Arkansas 6,324 57.8 4,070 37.2 493 4.5 61 0.6 
Colorado 2,837 56.0 1,876 37.0 187 3.7 165 3.3 
District of Columbia 169 25.6 450 68.1 33 5.0 9 1.4 
Florida 2,268 93.1 75 3.1 76 3.1 16 0.7 
Illinois 17,279 58.3 10,414 35.1 1,565 5.3 398 1.3 
Louisiana 8,857 75.6 2,239 19.1 515 4.4 100 0.9 
Maryland 11,530 54.3 6,934 32.6 2,466 11.6 310 1.5 
Massachusetts 13,509 78.0 3,165 18.3 519 3.0 120 0.7 
Mississippi 11,957 84.5 1,549 10.9 456 3.2 186 1.3 
New Jersey 16,348 56.2 11,026 37.9 1,596 5.5 142 0.5 
New Mexico 2,721 76.2 723 20.2 98 2.7 30 0.8 
New York 1,558 57.8 1,031 38.2 72 2.7 35 1.3 
Ohio 17,903 58.2 11,254 36.6 1,298 4.2 315 1.0 
Rhode Island 2,439 46.6 2,611 49.9 152 2.9 30 0.6 
Tennessee 10,705 63.4 5,780 34.2 309 1.8 83 0.5 
All States 138,600 62.9 68,723 31:2 10,650 4.8 2,222 1.0 


Note. Percentages are based on the number of valid responses. Missing = 963. 
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Table 21, Student Survey ELA CBT (EOY): Q10. What did you use to take this test? 


Desktop Computer Laptop Computer Tablet with a Keyboard Tablet without Keyboard 
State n % n % n % n % 
Arizona 8,899 65.3 4,036 29.6 522 3.8 165 1.2 
Arkansas 5,066 60.6 2,981 35.6 246 2.9 71 0.8 
Colorado 2,551 47.5 2,264 42.2 481 9.0 69 1.3 
District of Columbia 614 30.1 1,330 65.1 76 3.7 23 1.1 
Florida 1,913 91.6 103 4.9 52 2 21 1.0 
Illinois 13,180 56.9 8,463 36.6 1,158 5.0 343 1.5 
Louisiana 5,016 71.4 1,492 21.3 407 5.8 106 1.5 
Maryland 13,217 77.8 3,182 18.7 470 2.8 130 0.8 
Massachusetts 10,096 54.4 5,863 31.6 2,246 12.1 346 1.9 
Mississipp1 8,198 82.7 1,262 12.7 294 3.0 159 1.6 
New Jersey 11,006 52.0 8,397 39.7 1,529 7.2 218 1.0 
New Mexico 1,507 84.0 197 11.0 76 4.2 15 0.8 
New York 2,193 61.5 1,155 32.4 167 4.7 53 1.5 
Ohio 11,702 57.7 7,405 36.5 905 4.5 284 1.4 
Rhode Island 1,202 46.2 1,295 49.8 98 3.8 6 0.2 
Tennessee 2,381 62.9 1,295 34.2 80 2.1 31 0.8 
All States 98,741 61.6 50,720 31.6 8,807 5.5 2,040 1.3 
Note. Percentages are based on the number of valid responses. Missing = 512. 
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Table 22. Student Survey ELA CBT (PBA): Q11. How many times did you practice on a computer or tablet to get 
ready for this test? 


Never Once More than Once 

State n % n % n % 
Arizona 9,520 50.8 6,224 33.2 2,998 16.0 
Arkansas 3,386 30.9 4,732 43.2 2,826 25.8 
Colorado 2,406 47.5 1,601 31.6 1,057 20.9 
District of 226 34.1 241 36.4 195 29.5 
Columbia 

Florida 1,248 51.3 702 28.8 181 19.9 
Illinois 13,251 44.7 11,290 38.1 5,108 17.2 
Louisiana 5,234 44.7 3,975 34.0 2,496 21.3 
Maryland 5,835 33.7 6,569 38.0 4,892 28.3 
Massachusetts 8,156 38.4 8,658 40.8 4,423 20.8 
Mississippi 6,086 43.0 4,664 33.0 3,394 24.0 
New Jersey 11,076 38.1 11,326 38.9 6,703 23.0 
New Mexico 1,579 44.2 1,304 36.5 687 19.2 
New York 1,451 53.8 737 27.3 510 18.9 
Ohio 17,511 56.8 9,194 29.8 4,105 13.3 
Rhode Island 2,228 42.6 1,852 35.4 1,151 22.0 
Tennessee 8,542 50.6 5,256 31.2 3,073 18.2 
All States 97,735 44.4 78,325 35.6 44,102 20.0 


Note. Percentages are based on the number of valid responses. Missing = 996. 


Table 23. Student Survey ELA CBT (EOY): Q11. How many times did you practice on a computer or tablet to get 
ready for this test? 


Never Once More than Once 

State n % n % n % 

Arizona 7,158 52.6 4,410 32.4 2,044 15.0 
Arkansas 2,756 33.0 3,868 46.3 1,736 20.8 
Colorado 3,089 57.6 1,363 25.4 912 17.0 
District of Columbia 927 45.5 606 29.7 506 24.8 
Florida 895 42.9 727 34.8 465 22.3 
Illinois 10,364 44.8 8,855 38.3 3,923 17.0 
Louisiana 3,216 45.8 2,296 32.7 1,516 21.6 
Maryland 6,391 37.6 6,130 36.1 4,463 26.3 
Massachusetts 7,837 42.2 7,365 39.7 3,348 18.0 
Mississippi 4,863 49.1 2,752 27.8 2,295 23.2, 
New Jersey 8,393 39.7 8,033 38.0 4,717 22.3 
New Mexico 787 43.9 685 38.2 320 17.9 
New York 1,584 44.5 1,340 37.7 635 17.8 
Ohio 11,338 55.8 5,847 28.8 3,132 15.4 
Rhode Island 975 37.5 956 36.7 672 25.8 
Tennessee 2,020 53.4 1,042 27.6 718 19.0 
All States 72,593 45.3 56,275 35.1 31,402 19.6 


Note. Percentages are based on the number of valid responses. Missing = 550. 
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Table 24. Student Survey ELA CBT (PBA): Q12. Was it easy to use the highlighter? 


Yes No Did Not Use Highlighter 

State n % n % n % 

Arizona 12,259 65.4 901 4.8 5,581 29.8 
Arkansas 7,111 65.0 409 3.7 3,419 31.3 
Colorado 2,870 56.8 310 6.1 1,872 37.1 
District of Columbia 489 73.9 50 7.6 123 18.6 
Florida 1,631 67.0 116 4.8 687 28.2 
Illinois 17,056 57.6 1,225 4.1 11,352 38.3 
Louisiana 7,852 67.1 484 4.1 3,362 28.7 
Maryland 10,559 61.1 901 5.2 5,832 33.7 
Massachusetts 13,158 62.0 1,320 6.2 6,761 31.8 
Mississippi 9,302 65.8 490 3.5 4,347 30.7 
New Jersey 17,728 60.9 1,314 4.5 10,054 34.6 
New Mexico 2,440 68.4 202 5.7 923 25.9 
New York 1,745 64.7 160 5.9 790 29.3 
Ohio 17,472 56.7 1,369 4.4 11,955 38.8 
Rhode Island 3,398 65.0 233 4.5 1,599 30.6 
Tennessee 9,932 58.9 774 4.6 6,160 36.5 
All States 135,002 61.3 10,258 4.7 74,817 34.0 


Note. Percentages are based on the number of valid responses. Missing = 1081. 


Table 25. Student Survey ELA CBT (EOY): Q12. Was it easy to use the highlighter? 


Yes No Did Not Use Highlighter 

State n % n % n % 

Arizona 9,094 66.8 824 6.1 3,690 27.1 
Arkansas 5,447 65.2 424 5.1 2,485 29.7 
Colorado 2,895 54.0 303 5.6 2,166 40.4 
District of Columbia 1,351 66.2 177 8.7 512 25.1 
Florida 1,395 66.9 127 6.1 563 27.0 
Illinois 13,732 59.3 1,148 5.0 8,269 35.7 
Louisiana 4,663 66.4 426 6.1 1,930 27.5 
Maryland 10,671 62.8 989 5.8 5,320 31.3 
Massachusetts 10,589 57.1 1,358 73 6,591 35.6 
Mississippi 6,557 66.2 508 5.1 2,838 28.7 
New Jersey 12,578 59.5 1,190 5.6 7,371 34.9 
New Mexico 1,271 70.7 92 5.1 434 24.2 
New York 2,255 63.3 260 7.3 1,047 29.4 
Ohio 11,940 58.8 1,389 6.8 6,973 34.3 
Rhode Island 1,705 65.6 119 4.6 776 29.8 
Tennessee 2,156 57.0 277 73 1,352 35:7 
All States 98,299 61.3 9,611 6.0 52,317 32.7 


Note. Percentages are based on the number of valid responses. Missing = 593. 
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Table 26. Student Survey ELA CBT (PBA): Q13. Was it easy to type your answers? 


Yes No Did Not Type Answers 
State n % n % n % 
Arizona 16,143 86.3 2,039 10.9 522 2.8 
Arkansas 9,808 89.7 901 8.2 220 2.0 
Colorado 4,332 85.6 658 13.0 68 1.3 
District of Columbia 592 89.3 65 9.8 6 0.9 
Florida 2,121 87.2 284 11.7 26 1.1 
Illinois 25,916 87.6 3,181 10.8 492 1.7 
Louisiana 10,213 87.4 1,242 10.6 232 2.0 
Maryland 14,991 86.8 1,839 10.7 437 2.5 
Massachusetts 18,491 87.2 2,421 11.4 298 1.4 
Mississippi 12,243 86.7 1,432 10.1 452 3.2 
New Jersey 25,407 87.4 3,282 11.3 382 1.3 
New Mexico 3,228 90.5 301 8.4 39 1.1 
New York 2,220 82.5 391 14.5 79 2.9 
Ohio 26,902 87.4 3,401 11.1 471 1.5 
Rhode Island 4,533 86.7 617 11.8 77 1.5 
Tennessee 14,743 87.5 1,719 10.2 393 2.3 
All States 191,883 87.3 23,773 10.8 4,194 1.9 


Note. Percentages are based on the number of valid responses. Missing = 1308. 


Table 27. Student Survey ELA CBT (EOY): Q13. Was it easy to type your answers? 


Yes No Did Not Type Answers 

State n % n % n % 

Arizona 5,809 42.7 654 4.8 7,140 52.5 
Arkansas 3,436 41.1 304 3.6 4,611 55.2 
Colorado 1,988 37.1 237 4.4 3,138 58.5 
District of Columbia 870 42.6 116 Sef 1,054 51.7 
Florida 948 45.5 97 4.7 1,038 49.8 
Illinois 8,480 36.6 916 4.0 13,746 59.4 
Louisiana 3,058 43.5 335 4.8 3,629 51.7 
Maryland 7,151 42.1 704 4.1 9,126 53.7 
Massachusetts 6,988 37.7 960 5.2 10,584 57.1 
Mississippi 3,865 39.0 462 4.7 5,575 56.3 
New Jersey 7,544 35.7 873 4.1 12,716 60.2 
New Mexico 722 40.4 67 3.7 1,000 55.9 
New York 1,206 33.9 188 5.3 2,166 60.8 
Ohio 6,935 34.2 1,058 a2. 12,301 60.6 
Rhode Island 1,163 44.7 108 4.1 1,333 51.2 
Tennessee 1,505 39.8 173 4.6 2,102 55.6 
All States 61,668 38.5 7,252 4.5 91,259 57.0 


Note. Percentages are based on the number of valid responses. Missing = 641. 
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Table 28. Student Survey ELA CBT (PBA): Q14. Was it easy to make pictures or words bigger or smaller? 


Yes No Did Not Change Size 
State n % n % n % 
Arizona 6,248 33.4 1,621 8.7 10,838 57.9 
Arkansas 3,455 31.6 742 6.8 6,733 61.6 
Colorado 1,356 26.8 531 10.5 3,169 62.7 
District of Columbia 287 43.4 58 8.8 317 47.9 
Florida 815 33.6 213 8.8 1,398 57.6 
Illinois 8,080 27.3 2,332 7.9 19,202 64.8 
Louisiana 4,246 36.4 907 7.8 6,525 55.9 
Maryland 5,234 30.3 1,647 9.5 10,401 60.2 
Massachusetts 6,467 30.5 1,962 9.2 12,787 60.3 
Mississippi 4,630 32.8 913 6.5 8,587 60.8 
New Jersey 7,879 27.1 2,493 8.6 18,706 64.3 
New Mexico 1,257 35.3 343 9.6 1,963 55.1 
New York 922 34.3 280 10.4 1,485 55.3 
Ohio 7,705 25.0 2,413 7.8 20,653 67.1 
Rhode Island 1,595 30.5 540 10.3 3,095 59.2 
Tennessee 5,311 31.5 1,243 7A 10,293 61.1 
All States 65,487 29.8 18,238 8.3 136,152 61.9 


Note. Percentages are based on the number of valid responses. Missing = 1281. 


Table 29. Student Survey ELA CBT (EOY): Q14. Was it easy to make pictures or words bigger or smaller? 


Yes No Did Not Change Size 
State n % n % n % 
Arizona 3,793 27.9 998 73 8,804 64.8 
Arkansas 2,233 26.7 527 6.3 5,595 67.0 
Colorado 1,105 20.6 409 7.6 3,844 71.7 
District of Columbia 582 28.5 172 8.4 1,287 63.1 
Florida 566 27.2 162 7.8 1,356 65.1 
Illinois 5,424 23.4 1,501 6.5 16,209 70.1 
Louisiana 2,177 31.1 520 74 4,313 61.5 
Maryland 4,464 26.3 1,350 7.9 11,168 65.8 
Massachusetts 4,722 25.5 1,456 7.9 12,355 66.7 
Mississippi 2,644 26.7 641 6.5 6,617 66.8 
New Jersey 5,122 24.2 1,479 7.0 14,528 68.8 
New Mexico 631 35.2 104 5.8 1,060 59.1 
New York 872 24.5 254 71 2,433 68.4 
Ohio 4,789 23.6 1,616 8.0 13,880 68.4 
Rhode Island 609 23.5 227 8.7 1,761 67.8 
Tennessee 1,068 28.3 269 7A 2,439 64.6 
All States 40,801 25.5 11,685 73 107,649 67.2 


Note. Percentages are based on the number of valid responses. Missing = 685. 
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Table 30. Student Survey ELA CBT (PBA): Q15. Was it easy to move back and forth between passages or stories? 


Did Not Move 
Yes No Between Passages or Stories 

State n % n % n % 

Arizona 16,224 86.9 1,300 7.0 1,150 6.2 
Arkansas 9,747 89.2 672 6.2 507 4.6 
Colorado 4,407 87.2 354 7.0 294 5.8 
District of Columbia 584 88.1 46 6.9 33 5.0 
Florida 2,166 89.3 156 6.4 103 4.2 
Illinois 25,927 87.6 1,946 6.6 1,723 5.8 
Louisiana 10,123 86.7 842 732 709 6.1 
Maryland 15,138 87.7 1,168 6.8 956 55 
Massachusetts 18,448 87.1 1,629 7.7 1,108 5,2 
Mississippi 12,469 88.3 825 5.8 825 5.8 
New Jersey 25,595 88.1 2,178 ha 1,272 4.4 
New Mexico 3,147 88.4 240 6.7 173 4.9 
New York 2,267 84.2 247 9.2 177 6.6 
Ohio 26,310 85.5 2,304 12 2,149 7.0 
Rhode Island 4,607 88.1 379 7.3 241 4.6 
Tennessee 14,728 87.5 1,007 6.0 1,098 6.5 
All States 191,887 87.3 15,293 7.0 12,518 Sul 


Note. Percentages are based on the number of valid responses. Missing = 1460. 


Table 31. Student Survey ELA CBT (EOY): Q15. Was it easy to move back and forth between passages or stories? 


Did Not Move 
Yes No Between Passages or Stories 

State n % n % n % 

Arizona 11,163 82.2 1,004 7.4 1,416 10.4 
Arkansas 6,944 83.2 537 6.4 866 10.4 
Colorado 4,358 81.5 409 7.7 579 10.8 
District of Columbia 1,670 81.9 180 8.8 189 9.3 
Florida 1,823 87.6 124 6.0 135 6.5 
Illinois 18,831 81.4 1,560 6.7 2,733 11.8 
Louisiana 5,710 81.4 567 8.1 742 10.6 
Maryland 14,198 83.7 1,105 6.5 1,670 9.8 
Massachusetts 14,864 80.3 1,600 8.6 2,052 11.1 
Mississippi 8,206 82.9 676 6.8 1,017 10.3 
New Jersey 17,230 81.6 1,703 8.1 2,189 10.4 
New Mexico 1,544 86.4 100 5.6 144 8.1 
New York 2,889 81.3 288 8.1 377 10.6 
Ohio 15,516 76.5 1,920 9.5 2,850 14.0 
Rhode Island 2,159 83.1 175 6.7 265 10.2 
Tennessee 3,044 80.7 296 78 434 11.5 
All States 130,149 81.3 12,244 7.7 17,658 11.0 


Note. Percentages are based on the number of valid responses. Missing = 769. 
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Table 32. Student Survey ELA CBT (PBA): Q16. Was it easy to find information in the passages or stories when 
answering questions? 


Did Not Move 
Yes No Between Passages or Stories 

State n % n % n % 
Arizona 13,998 75.0 3,994 21.4 682 3.7 
Arkansas 8,457 774 2,144 19.6 320 2.9 
Colorado 3,778 74.7 1,081 21.4 196 3.9 
District of 545 82.8 96 14.6 17 2.6 
Columbia 

Florida 1,953 80.6 399 16.5 71 2.9 
Illinois 22,068 74.6 6,442 21.8 1,084 3.7 
Louisiana 8,868 75.9 2,404 20.6 413 3.5 
Maryland 13,145 76.1 3,514 20.4 603 3.5 
Massachusetts 15,977 75.4 4,566 21.5 647 3.1 
Mississippi 11,141 78.9 2,572 18.2 414 2.9 
New Jersey 21,023 72.4 7,269 25.0 763 2.6 
New Mexico 2,771 77.9 690 19.4 98 2.8 
New York 2,012 74.7 568 21.1 113 4.2 
Ohio 22,329 72.6 7,320 23.8 1,090 35 
Rhode Island 3,942 75.5 1,150 22.0 127 2.4 
Tennessee 13,124 78.0 3,082 18.3 613 3.6 
All States 165,131 75.2 47,291 21.5 7,251 3.3 


Note. Percentages are based on the number of valid responses. Missing = 1485. 


Table 33. Student Survey ELA CBT (EOY): Q16. Was it easy to find information in the passages or stories when 
answering questions? 


Did Not Move 
Yes No Between Passages or Stories 

State n % n % n % 

Arizona 10,127 74.7 2,783 20.5 646 4.8 
Arkansas 6,302 75.5 1,698 20.3 344 4.1 
Colorado 4,035 75.4 1,108 20.7 210 3.9 
District of Columbia 1,568 77.1 387 19.0 80 3.9 
Florida 1,700 81.8 310 14.9 67 3.2 
Illinois 17,152 74.3 4,962 21.5 982 4.3 
Louisiana 5,277 75.3 1,396 19.9 332 4.7 
Maryland 13,014 76.7 3,276 19.3 682 4.0 
Massachusetts 13,519 73.1 4,243 22.9 742 4.0 
Mississippi 7,554 76.4 1,866 18.9 469 4.7 
New Jersey 15,222 72.1 5,128 24.3 760 3.6 
New Mexico 1,406 78.7 305 17.1 76 4.3 
New York 2,638 74.2 773 21.7 145 4.1 
Ohio 14,405 71.1 4,721 23.3 1,137 5.6 
Rhode Island 1,944 74.9 562 21.6 91 3.5 
Tennessee 2,823 74.8 756 20.0 193 5.1 
All States 118,686 74.2 34,274 21.4 6,956 4.3 


Note. Percentages are based on the number of valid responses. Missing = 904. 
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Table 34. Student Survey ELA CBT (PBA): Q17. Did any of the problems listed below happen during your test? (Select all that apply)’. 


Computer Stopped Computer Worked Difficulty Dragging or Difficulty Making Changes No Problems Occurred 
Working Slowly Moving Things on Screen to Answers During the Test 

State n % n % n % n % n % 

Arizona 3,151 16.7 4,259 22.6 3,429 18.2 2,238 11.9 10,744 56.9 
Arkansas 1,544 14.1 2,445 22.3 2,087 19.0 1,129 10.3 6,450 58.8 
Colorado 935 18.4 1,345 26.4 1,082 21.3 744 14.6 2,622 51.5 
District of Columbia 77 11.6 212 32.0 159 24.0 75 11.3 329 49.6 
Florida 873 35.7 826 33.8 368 15.0 269 11.0 987 40.4 
Illinois 4,292 14.4 6,836 23.0 6,159 20.7 3,500 11.8 16,954 56.9 
Louisiana 1,781 15.2 2,221 18.9 1,918 16.3 1,316 11.2 7,271 61.9 
Maryland 3,350 19.3 4,623 26.6 3,178 18.3 2,266 13.0 9,371 53.9 
Massachusetts 3,297 15.4 4,745 22.2 4,602 21.6 2,954 13.8 11,904 55.8 
Mississipp1 2,293 16.2 2,923 20.6 2,062 14.5 1,360 9.6 8,726 61.5 
New Jersey 5,127 17.6 7,173 24.6 6,750 23.1 3,923 13.4 15,412 52.8 
New Mexico 678 18.9 817 22.8 675 18.8 383 10.7 2,045 57.1 
New York 420 15.5 462 17.0 583 21.5 373 13.7 1,579 58.2 
Ohio 4,182 13.5 6,782 21.9 7,086 22.9 3,779 12.2 17,598 56.9 
Rhode Island 1,092 20.8 1,600 30.5 1,114 21.2 737 14.0 2,671 50.9 
Tennessee 3,397 20.0 4,513 26.5 3,118 18.3 1,898 11.2 9,198 54.1 
All States 36,489 16.5 51,782 23.4 44,370 20.1 26,944 12.2 123,861 56.0 


“ Percentages reported for “select all that apply” should be interpreted with caution, as it is not possible to differentiate “missing” (i.e., skipped) vs. “not selected” 
responses. 
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Table 35. Student Survey ELA CBT (EOY): Q17. Did any of the problems listed below happen during your test? (Select all that apply)’. 


Computer Stopped Computer Worked Difficulty Dragging or Difficulty Making Changes No Problems Occurred 
Working Moving Things on Screen to Answers During the Test 

State n % n % n % n % n % 

Arizona 1,406 10.3 2,305 16.9 2,951 21.6 1,349 9.9 8,109 59.3 
Arkansas 890 10.6 1,448 17.3 1,771 21.2 749 8.9 5,081 60.7 
Colorado 459 8.5 1,010 18.8 1,339 24.9 622 11.6 3,093 57.6 
District of Columbia 270 13.1 427 20.7 493 23.9 225 10.9 1,096 53.1 
Florida 502 23.9 366 17.4 273 13.0 199 9.5 1,145 54.6 
Illinois 1,837 7.9 3,673 15.8 5,462 23.6 2,086 9.0 14,109 60.8 
Louisiana 831 11.8 1,147 16.3 1,277 18.1 620 8.8 4,406 62.6 
Maryland 2,113 12.4 3,269 19.2 3,046 17.9 1,620 9.5 10,349 60.8 
Massachusetts 1,790 9.6 3,479 18.7 4,516 24.3 1,884 10.1 10,828 58.2 
Mississippi 880 8.9 1,334 13.4 1,809 18.2 895 9.0 6,596 66.4 
New Jersey 2,083 9.8 3,918 18.5 5,294 24.9 2,186 10.3 12,052 56.8 
New Mexico 169 9.4 252 14.0 325 18.1 154 8.6 1,160 64.4 
New York 389 10.9 475 13.3 733 20.5 400 11.2 2,217 61.9 
Ohio 1,743 8.5 3,809 18.7 5,526 27.1 2,256 11.0 11,466 56.2 
Rhode Island 311 11.9 695 26.6 521 20.0 228 8.7 1,444 55.4 
Tennessee 455 12.0 695 18.3 925 24.3 390 10.3 2,126 55.9 
All States 16,128 10.0 28,302 17.6 36,261 22.5 15,863 9.9 95,277 59.2 


“ Percentages reported for “select all that apply” should be interpreted with caution, as it is not possible to differentiate “missing” (i.e., skipped) vs. “not selected” 


responses. 
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Appendix N — Item-level Results from the Student Survey for 
English Language Arts: Paper-based Test Administration 


Table 1. Student Survey Responses by State (ELA PBT) 


Performance-Based Assessment End-of-Y ear Assessment 
State n % of Total Responses n % of Total Responses 
Arizona 8,856 13.5 8,254 15.5 
Arkansas 5,292 8.1 4,236 8.0 
Colorado 1,909 2.9 1,892 3.6 
District of Columbia 39 0.1 268 0.5 
Florida 79 0.1 -- -- 
Illinois 10,051 15.3 7,473 14.0 
Louisiana 209 0.3 378 0.7 
Maryland 7,274 11.1 7,239 13.6 
Massachusetts 5,889 9.0 5,500 10.3 
Mississippi 8,020 12.2 7,978 15.0 
New Jersey 1 0.0 -- -- 
New Mexico 18 0.0 -- -- 
New York 1,365 2.1 1,064 2.0 
Ohio 8,216 12.5 6,468 12.1 
Rhode Island 1,961 3.0 966 1.8 
Tennessee 6,394 9.8 1,563 2.9 
All States 65,573 100.0 53,279 100.0 
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Table 2. Student Survey ELA PBT (PBA): Q1. Did you understand all of the directions read by the person who gave 


you the test? 


Yes No 

State n % n % 

Arizona 8,173 92.5 667 75 
Arkansas 5,067 95.8 221 4.2 
Colorado 1,769 92.8 137 ioe 
District of Columbia 36 94.7 2 53 
Florida 75 94.9 4 5.1 
Illinois 9,516 94.8 519 5.2 
Louisiana 201 96.6 7 3.4 
Maryland 6,968 96.0 289 4.0 
Massachusetts 5,572 94.8 304 5.2 
Mississippi 7,723 96.5 280 3.5 
New Jersey 1 100.0 0 0.0 
New Mexico 18 100.0 0 0.0 
New York 1,298 95.7 58 4.3 
Ohio 7,907 96.5 291 35 
Rhode Island 1,876 95.9 80 41 
Tennessee 6,164 96.7 212 3.3 
All States 62,364 95.3 3,071 4.7 


Note. Percentages are based on the number of valid responses. Missing = 138. 


Table 3. Student Survey ELA PBT (EOY): Q1. Did you understand all of the directions read by the person who gave 


you the test? 


Yes No 
State n % n % 
Arizona 7,318 88.8 926 11.2 
Arkansas 3,846 91.0 381 9.0 
Colorado 1,689 89.6 195 10.4 
District of Columbia 251 93.7 17 6.3 
Illinois 6,782 90.9 676 9.1 
Louisiana 347 91.8 31 8.2 
Maryland 6,493 89.9 733 10.1 
Massachusetts 5,024 91.5 465 8.5 
Mississippi 7,448 93.6 511 6.4 
New York 973 91.6 89 8.4 
Ohio 5,811 90.0 645 10.0 
Rhode Island 893 92.5 72 75 
Tennessee 1,427 91.7 130 8.3 
All States 48,302 90.8 4,871 9.2 
Note. Percentages are based on the number of valid responses. Missing = 106. 
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Table 4. Student Survey ELA PBT (PBA): Q2. How often was it hard to understand the directions for the questions 
on this test? 


Almost Always Most of the Time Some of the Time Almost Never 
State n % n % n % n % 
Arizona 514 5.8 1,412 16.0 4,624 52.3 2,285 25.9 
Arkansas 279 5.3 654 12.4 2,443 46.2 1,908 36.1 
Colorado 88 4.6 219 11.5 992 52.1 606 31.8 
District of Columbia 2, 5.4 6 16.2 23 62.2 6 16.2 
Florida 10 12.7 5 6.3 38 48.1 26 32.9 
Illinois 417 4.2 1,192 11.9 5,198 51.8 3,224 32.1 
Louisiana 10 4.8 42 20.2 82 39.4 74 35.6 
Maryland 337 4.6 876 12.1 3,537 48.7 2,510 34.6 
Massachusetts 322 5.5 804 13.7 2,915 49.6 1,832 31.2 
Mississipp1 364 4.6 981 12.3 3,860 48.3 2,788 34.9 
New Jersey 0 0.0 0 0.0 0 0.0 1 100.0 
New Mexico 0 0.0 2 11.1 9 50.0 7 38.9 
New York 68 5.0 229 16.9 672 49.6 387 28.5 
Ohio 394 4.8 922 11.3 4,011 49.0 2,866 35.0 
Rhode Island 121 6.2 251 12.8 899 46.0 685 35.0 
Tennessee 326 5.1 712 11.2 3,053 47.9 2,283 35.8 
All States 3,252 5.0 8,307 12.7 32,356 49.5 21,488 32.9 


Note. Percentages are based on the number of valid responses. Missing = 170. 


Table 5. Student Survey ELA PBT (EOY): Q2. How often was it hard to understand the directions for the questions 
on this test? 


Almost Always Most of the Time Some of the Time Almost Never 
State n % n % n % n % 
Arizona 490 5.9 1,226 14.9 4,183 50.7 2,347 28.5 
Arkansas 231 5.5 440 10.4 1,829 43.2 1,729 40.9 
Colorado 108 5.7 239 12.7 903 47.9 637 33.8 
District of Columbia 10 3.7 38 14.2 116 43.4 103 38.6 
Illinois 380 5.1 837 11.2 3,585 48.1 2,652 35.6 
Louisiana 19 5.0 55 14.6 175 46.4 128 34.0 
Maryland 356 4.9 962 13.3 3,432 47.5 2,471 34.2 
Massachusetts 265 4.8 688 12.5 2,585 47.1 1,948 35.5 
Mississipp1 423 5.3 960 12.1 3,652 45.9 2,917 36.7 
New York 61 5.8 136 12.9 523 49.5 337 31.9 
Ohio 332 5.1 776 12.0 3,158 49.0 2,184 33.9 
Rhode Island 53 5.5 121 1235 423 43.8 368 38.1 
Tennessee 110 7.1 215 13.8 709 45.5 523 33.6 
All States 2,838 5.3 6,693 12.6 25,273 47.6 18,344 34.5 


Note. Percentages are based on the number of valid responses. Missing = 131. 
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Table 6. Student Survey ELA PBT (PBA): Q3. How many questions asked you about things you have not learned in 


school this year? 


All of Them Most of Them Few of Them None of Them 
State n % n % n % n % 
Arizona 229 2.6 935 10.6 4,479 50.7 3,189 36.1 
Arkansas 171 3.2 501 9.5 2,575 48.8 2,034 38.5 
Colorado 34 1.8 178 9.4 951 50.0 738 38.8 
District of Columbia 1 eh 4 10.8 19 51.4 13 35.1 
Florida 7 8.9 20 25.3 28 35.4 24 30.4 
Illinois 175 1.7 1,013 10.1 5,701 56.9 3,136 31.3 
Louisiana 6 2.9 17 8.2 96 46.2 89 42.8 
Maryland 206 2.8 698 9.6 3,821 52.6 2,534 34.9 
Massachusetts 148 2:5 562 9.6 3,008 51.2 2,153 36.7 
Mississippi 230 2.9 593 7A 3,433 43.0 3,736 46.7 
New Jersey 0 0.0 0 0.0 1 100.0 0 0.0 
New Mexico 0 0.0 0 0.0 9 50.0 9 50.0 
New York 66 4.9 155 11.4 650 47.9 485 35.8 
Ohio 231 2.8 821 10.0 4,253 52.0 2,881 35.2 
Rhode Island 56 2.9 208 10.6 1,014 51.9 676 34.6 
Tennessee 173 27 514 8.1 3,061 48.1 2,620 41.1 
All States 1,733 2.7 6,219 9.5 33,099 50.6 24,317 37.2 


Note. Percentages are based on the number of valid responses. Missing = 205. 


Table 7. Student Survey ELA PBT (EOY): Q3. How many questions asked you about things you have not learned in 


school this year? 


All of Them Most of Them Few of Them None of Them 
State n % n % n % n % 
Arizona 240 2.9 921 11.2 4,438 53.9 2,635 32.0 
Arkansas 158 3.7 412 9.8 2,123 50.2 1,532 36.3 
Colorado 82 4.4 229 12.2 946 50.2 627 33.3 
District of Columbia 11 4.2 24 9.1 101 38.1 129 48.7 
Illinois 176 2.4 859 11.5 4,279 57.4 2,136 28.7 
Louisiana 7 1.9 38 10.1 183 48.4 150 39.7 
Maryland 197 2.7 778 10.8 3,803 52.7 2,439 33.8 
Massachusetts 135 2S 486 8.9 2,782 50.8 2,073 37.9 
Mississippi 252 3.2 629 7.9 3,549 44.6 3,523 44,3 
New York 48 4.6 117 11.1 533 50.6 356 33.8 
Ohio 184 2.9 689 10.7 3,436 53.3 2,137 33.2 
Rhode Island 4l 43 125 13.0 486 50.4 312 32.4 
Tennessee 50 3.2 177 11.4 861 55.4 465 29.9 
All States 1,581 3.0 5,484 10.3 27,520 51.8 18,514 34.9 
Note. Percentages are based on the number of valid responses. Missing = 180. 
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Table 8. Student Survey ELA PBT (PBA): Q4. How difficult was this test? 


Easier than School Work Same as School Work Harder than School Work 
State n % n % n % 
Arizona 1,013 11.5 4,470 50.7 3,327 37.8 
Arkansas 758 14.3 2,894 54.8 1,632 30.9 
Colorado 343 18.1 1,035 54.5 521 27.4 
District of Columbia 2 5.4 19 51.4 16 43.2 
Florida 3 3.8 38 48.1 38 48.1 
Illinois 1,219 12.2 5,019 50.1 3,780 37.7 
Louisiana 25 12.0 86 41.3 97 46.6 
Maryland 966 13.3 4,086 56.3 2,203 30.4 
Massachusetts 686 11.7 2,789 47.6 2,386 40.7 
Mississippi 1,235 15.5 4,555 57.0 2,203 27.6 
New Jersey 0 0.0 1 100.0 0 0.0 
New Mexico 4 22.2 12 66.7 2 11.1 
New York 161 11.9 847 62.6 345 25:5 
Ohio 986 12.1 4,246 51.9 2,947 36.0 
Rhode Island 168 8.6 919 47.1 866 44,3 
Tennessee 783 12.3 3,443 54.1 2,136 33.6 
All States 8,352 12.8 34,459 52.8 22,499 34.4 
Note. Percentages are based on the number of valid responses. Missing = 263. 
Table 9, Student Survey ELA PBT (EOY): Q4. How difficult was this test? 

Easier than School Work Same as School Work Harder than School Work 
State n % n % n % 
Arizona 1,015 12.3 4,159 50.5 3,062 37.2 
Arkansas 667 15.8 2,343 55.5 1,212 28.7 
Colorado 351 18.7 1,022 54.4 506 26.9 
District of Columbia 27 10.1 170 63.7 70 26.2 
Illinois 898 12.0 3,829 51.4 2,727 36.6 
Louisiana 52 13.8 207 55.1 117 31.1 
Maryland 868 12.0 3,977 55.1 2,370 32.8 
Massachusetts 766 14.0 2,887 52.7 1,823 33.3 
Mississippi 1,037 13.1 4,581 a7 2,327 29.3 
New York 136 12.9 613 58.2 305 28.9 
Ohio 679 10.5 3,308 51.3 2,457 38.1 
Rhode Island 115 12.0 433 45.1 412 42.9 
Tennessee 167 10.8 775 49.9 611 39.3 
All States 6,778 12.8 28,304 53.3 17,999 33.9 
Note. Percentages are based on the number of valid responses. Missing = 198. 
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Table 10. Student Survey ELA PBT (PBA): Q5. Did you have enough time to finish this test? 


Finished Very Early _ Finished On Time Rushed to Finish Did Not Finish 
State n % n % n % n % 
Arizona 2,596 29.4 5,116 58.0 723 8.2 384 4.4 
Arkansas 2,533 48.0 2,469 46.8 196 3.1 76 1.4 
Colorado 844 44.3 930 48.8 110 5.8 22 1.2 
District of Columbia 10 27.0 18 48.6 3 8.1 6 16.2 
Florida 37 46.8 35 44.3 4 5.1 3 3.8 
Illinois 4,657 46.5 5,014 50.0 266 2.7 87 0.9 
Louisiana 66 31.6 115 55.0 17 8.1 11 5.3 
Maryland 3,725 51.3 3,201 44.1 213 2.9 118 1.6 
Massachusetts 1,932 32.9 3,276 55.8 474 8.1 184 3.1 
Mississipp1 2,957 37.0 4,584 57.3 310 3.9 147 1.8 
New Jersey 1 100.0 0 0.0 0 0.0 0 0.0 
New Mexico 0 0.0 16 88.9 2 11.1 0 0.0 
New York 505 37.3 697 51.4 80 5.9 73 5.4 
Ohio 3,875 47.3 3,856 47.1 302 331 157 1.9 
Rhode Island 701 35.8 1,140 58.3 85 4.3 31 1.6 
Tennessee 2,858 44.9 3,116 48.9 232 3.6 161 2 
All States 27,297 41.8 33,583 51.4 3,017 4.6 1,460 2.2 


Note. Percentages are based on the number of valid responses. Missing = 216. 


Table 11. Student Survey ELA PBT (EOY): Q5. Did you have enough time to finish this test? 


Finished Very Early _ Finished On Time Rushed to Finish Did Not Finish 
State n % n % n % n % 
Arizona 3,029 36.8 4,519 54.9 525 6.4 162 2.0 
Arkansas 2,325 55.0 1,708 40.4 154 3.6 38 0.9 
Colorado 907 48.1 812 43.1 126 6.7 39 2.1 
District of Columbia 67 25.4 147 55.7 13 4.9 37 14.0 
Illinois 3,791 50.8 3,409 45.7 205 2.7 55 0.7 
Louisiana 102 27.0 247 65.3 15 4.0 14 331 
Maryland 3,465 48.0 3,364 46.6 271 3.8 114 1.6 
Massachusetts 2,217 40.4 2,880 52.5 294 5.4 96 1.7 
Mississipp1 2,865 36.1 4,587 57.7 346 4.4 147 1.9 
New York 487 46.1 463 43.8 67 6.3 40 3.8 
Ohio 2,706 42.0 3,311 51.3 292 4.5 140 2.2 
Rhode Island 402 41.7 524 54.4 29 3.0 8 0.8 
Tennessee 748 48.1 735 47.3 54 3.5 18 1.2 
All States 23,111 43.5 26,706 50.3 2,391 4.5 908 1.7 


Note. Percentages are based on the number of valid responses. Missing = 163. 
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Table 12. Student Survey ELA PBT (PBA): Q6. How often do you use a computer or tablet at home? 


A Couple Times A 
Every Day A Few Times a Week Month or Less Do Not Have at Home 

State n % n % n % n % 

Arizona 4,078 46.3 2,823 32.0 1,127 12.8 786 8.9 
Arkansas 2,757 52.3 1,332 25.3 593 11.2 593 11.2 
Colorado 892 46.9 673 35.4 216 11.4 120 6.3 
District of Columbia 19 51.4 10 27.0 5 13.5 3 8.1 
Florida 57 72.2 8 10.1 9 11.4 5 6.3 
Illinois 5,772 57.6 2,849 28.4 927 9.3 469 4.7 
Louisiana 72 34.4 73 34.9 28 13.4 36 17.2 
Maryland 3,738 51.6 2,300 31.7 798 11.0 412 5.7 
Massachusetts 3,310 56.4 1,846 31.5 479 8.2 234 4.0 
Mississipp1 3,777 47.3 2,279 28.5 1,017 12.7 919 11.5 
New Jersey 1 100.0 0 0.0 0 0.0 0 0.0 
New Mexico 6 33.3 6 33.3 3 16.7 3 16.7 
New York 606 44.8 442 32.7 180 13.3 124 9.2 
Ohio 4,641 56.7 2,289 28.0 749 9.2 503 6.1 
Rhode Island 1,131 57.9 606 31.0 134 6.9 83 4.2 
Tennessee 3,394 53.4 1,608 25.3 732 11.5 627 9.9 
All States 34,251 52.4 19,144 29.3 6,997 10.7 4,917 75 


Note. Percentages are based on the number of valid responses. Missing = 264. 


Table 13. Student Survey ELA PBT (EOY): Q6. How often do you use a computer or tablet at home? 


A Couple Times A 
Every Day A Few Times a Week Month or Less Do Not Have at Home 

State n % n % n % n % 

Arizona 3,825 46.5 2,652 32.2 1,051 12.8 705 8.6 
Arkansas 2,151 50.9 1,076 25.5 513 12.1 485 11.5 
Colorado 1,019 54.2 552 29.4 202 10.8 106 5.6 
District of Columbia 104 39.2 96 36.2 45 17.0 20 75 
Illinois 4,113 55.2 2,129 28.6 772 10.4 435 5.8 
Louisiana 163 43.5 123 32.8 48 12.8 41 10.9 
Maryland 3,796 52.6 2,260 31.3 803 11.1 356 4.9 
Massachusetts 3,046 55.7 1,695 31.0 516 9.4 216 3.9 
Mississippi 3,764 47.4 2,266 28.5 1,131 14.2 781 9.8 
New York 498 47.2 334 31.6 146 13.8 78 7.4 
Ohio 3,474 53.9 1,787 27.7 710 11.0 472 73 
Rhode Island 488 50.6 334 34.6 92 9.5 51 5.3 
Tennessee 808 52.0 380 24.5 196 12.6 169 10.9 
All States 27,249 51.3 15,684 29.6 6,225 11.7 3,915 74 


Note. Percentages are based on the number of valid responses. Missing = 206. 
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Table 14. Student Survey ELA PBT (PBA): Q7. How often do you use a computer or tablet in school? 


A Couple Times A 
Every Day A Few Times a Week Month or Less Do Not Use in School 

State n % n % n % n % 

Arizona 1,165 13.2 2,959 33.6 3,581 40.6 1,110 12.6 
Arkansas 1,156 21.9 1,981 37.5 1,853 35.1 290 5.5 
Colorado 353 18.6 825 43.4 616 32.4 106 5.6 
District of Columbia 10 26.3 20 52.6 4 10.5 4 10.5 
Florida 9 11.4 10 12.7 54 68.4 6 7.6 
Illinois 1,745 17.4 3,927 39.2 3,913 39.0 436 44 
Louisiana 52 24.9 95 45.5 42 20.1 20 9.6 
Maryland 620 8.6 2,711 37.4 3,321 45.8 595 8.2 
Massachusetts 640 10.9 2,205 37.5 2,604 44.3 424 12 
Mississipp1 1,819 22.8 2,630 32.9 2,774 34.7 771 9.6 
New Jersey 0 0.0 1 100.0 0 0.0 0 0.0 
New Mexico 0 0.0 6 33.3 10 55.6 2 11.1 
New York 110 8.1 618 45.5 433 31.9 196 14.4 
Ohio 1,577 19.3 3,469 42.4 2,681 32.7 464 5.7 
Rhode Island 178 9.1 813 41.6 821 42.0 143 73 
Tennessee 881 13.8 2,413 37.9 2,706 42.5 362 5.7 
All States 10,315 15.8 24,683 37.8 25,413 38.9 4,929 75 


Note. Percentages are based on the number of valid responses. Missing = 233. 


Table 15. Student Survey ELA PBT (EOY): Q7. How often do you use a computer or tablet in school? 


A Couple Times A 
Every Day A Few Times a Week Month or Less Do Not Use in School 

State n % n % n % n % 

Arizona 1,243 15.1 2,916 35.4 3,143 38.2 926 11.3 
Arkansas 1,067 25.3 1,755 41.5 1,220 28.9 182 4.3 
Colorado 396 21.1 810 43.1 560 29.8 115 6.1 
District of Columbia 19 7.3 63 24.1 126 48.3 53 20.3 
Illinois 1,743 23.4 3,403 45.6 2,031 271.2 278 3.7 
Louisiana 36 9.6 188 50.3 97 25.9 53 14.2 
Maryland 592 8.2 2,856 39.6 3,315 45.9 455 6.3 
Massachusetts 551 10.1 2,221 40.5 2,368 43.2 338 6.2 
Mississippi 1,649 20.8 2,694 33.9 2,928 36.9 668 8.4 
New York 110 10.5 431 41.0 378 36.0 132 12.6 
Ohio 1,327 20.6 2,958 45.9 1,838 28.5 322 5.0 
Rhode Island 106 11.0 506 52.5 301 31.2 51 5.3 
Tennessee 223 14.4 641 41.4 610 39.4 75 48 
All States 9,062 17.1 21,442 40.4 18,915 35.6 3,648 6.9 


Note. Percentages are based on the number of valid responses. Missing = 212. 
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Table 16. Student Survey ELA PBT (PBA): Q8. When writing a story or essay, how often do you use a computer or 
tablet? 


All of the Time Most of the Time Some of the Time Never 

State n % n % n % n % 

Arizona 744 8.4 1,776 20.1 3,329 37.8 2,966 33.6 
Arkansas 326 6.2 822 15.6 2,125 40.3 2,004 38.0 
Colorado 149 7.8 438 23.0 798 41.9 518 242 
District of Columbia 1 23) 6 16.2 20 54.1 10 27.0 
Florida 10 12.8 16 20.5 28 35.9 24 30.8 
Illinois 1,036 10.3 2,418 24.1 4,000 39.9 2,560 25.6 
Louisiana 7 3.3 19 9.1 46 22.0 137 65.6 
Maryland 635 8.8 1,451 20.0 2,821 38.9 2,346 32.3 
Massachusetts 673 11.5 1,354 23.0 2,100 35.7 1,749 29.8 
Mississippi 767 9.6 1,515 19.0 3,022 37.8 2,684 33.6 
New Jersey 0 0.0 0 0.0 0 0.0 1 100.0 
New Mexico 1 5.6 4 22.2 9 50.0 4 22.2 
New York 94 6.9 194 14.3 527 38.9 540 39.9 
Ohio 775 9.5 1,713 20.9 3,324 40.6 2,374 29.0 
Rhode Island 262 13.4 507 26.0 681 34.9 503 25.8 
Tennessee 402 6.3 1,126 17.7 2,926 46.0 1,909 30.0 
All States 5,882 9.0 13,359 20.4 25,756 39.4 20,329 31.1 


Note. Percentages are based on the number of valid responses. Missing = 247. 


Table 17. Student Survey ELA PBT (EOY): Q8. When writing a story or essay, how often do you use a computer or 
tablet? 


All of the Time Most of the Time Some of the Time Never 

State n % n % n % n % 

Arizona 999 12.1 1,962 23.8 3,162 38.4 2,108 25.6 
Arkansas 338 8.0 864 20.4 1,728 40.9 1,299 30.7 
Colorado 321 17.1 539 28.6 685 36.4 337 17.9 
District of Columbia 35 13.2 59 22.3 102 38.5 69 26.0 
Illinois 1,051 14.1 2,105 28.3 2,873 38.6 1,420 19.1 
Louisiana 22 5.9 30 8.0 105 28.2 216 57.9 
Maryland 769 10.7 1,595 22.1 2,944 40.8 1,907 26.4 
Massachusetts 914 16.7 1,601 29.2 1,944 35.5 1,021 18.6 
Mississippi 1,091 13.7 1,889 23.8 3,039 38.3 1,926 24.2 
New York 101 9.6 184 17.4 395 37.4 376 35.6 
Ohio 802 12.5 1,806 28.0 2,548 39.6 1,285 20.0 
Rhode Island 141 14.6 255 26.5 387 40.1 181 18.8 
Tennessee 210 13.5 359 23.1 627 40.4 356 22.9 
All States 6,794 12.8 13,248 25.0 20,539 38.7 12,501 23.6 


Note. Percentages are based on the number of valid responses. Missing = 197. 
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Table 18. Student Survey ELA PBT (PBA): Q9. Would you rather take this test on paper OR on a computer or 


tablet? 
On Paper On a Computer or Tablet 

State n % n % 
Arizona 3,142 35.7 5,652 64.3 
Arkansas 1,734 32.9 3,529 67.1 
Colorado 695 36.6 1,202 63.4 
District of Columbia 4 10.5 34 89.5 
Florida 50 64.9 27 35.1 
Illinois 3,888 38.9 6,111 61.1 
Louisiana 93 44.7 115 55.3 
Maryland 2,062 28.5 5,176 71.5 
Massachusetts 2,139 36.5 3,727 63.5 
Mississippi 2,343 29.4 5,637 70.6 
New Jersey 1 100.0 0 0.0 
New Mexico 3 16.7 15 83.3 
New York 459 34.1 887 65.9 
Ohio 3,351 41.0 4,814 59.0 
Rhode Island 653 33.5 1,298 66.5 
Tennessee 2,289 36.1 4,052 63.9 
All States 22,906 35.1 42,276 64.9 


Note. Percentages are based on the number of valid responses. Missing = 391. 


Table 19. Student Survey ELA PBT (EOY): Q9. Would you rather take this test on paper OR on a computer or 


tablet? 

On Paper On a Computer or Tablet 
State n % n % 
Arizona 2,794 34.0 5,413 66.0 
Arkansas 1,423 33.7 2,795 66.3 
Colorado 710 37.9 1,161 62.1 
District of Columbia 39 14.7 226 85.3 
Illinois 3,116 41.9 4,324 58.1 
Louisiana 129 34.3 247 65.7 
Maryland 1,960 27.2 5,251 72.8 
Massachusetts 1,873 34.2 3,599 65.8 
Mississippi 2,507 31.6 5,422 68.4 
New York 369 35.0 684 65.0 
Ohio 2,433 37.8 4,002 62.2 
Rhode Island 398 41.3 566 58.7 
Tennessee 573 36.9 978 63.1 
All States 18,324 34.6 34,668 65.4 
Note. Percentages are based on the number of valid responses. Missing = 287. 
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Table 20. Student Survey ELA PBT (PBA): Q10. Was it easy to flip back and forth between pages to find the story 
or passage that belonged to a question? 


Yes No Did Not Flip Between Pages 

State n % n % n % 

Arizona 6,153 69.9 2,215 25.1 440 5.0 
Arkansas 3,773 71.5 1,257 23.8 250 4.7 
Colorado 1,291 67.9 479 25.2 132 6.9 
District of Columbia 19 50.0 16 42.1 3 7.9 
Florida 57 72.2 18 22.8 4 5.1 
Illinois 7,248 72.4 2,307 23.0 454 4.5 
Louisiana 160 76.6 37 17.7 12 5.7 
Maryland 5,091 70.2 1,861 25.7 300 4.1 
Massachusetts 4,045 68.9 1,577 26.9 251 4.3 
Mississippi 6,300 78.8 1,350 16.9 349 4.4 
New Jersey 1 100.0 0 0.0 0 0.0 
New Mexico 13 72.2 a) 27.8 0 0.0 
New York 975 72.0 316 23.3 63 4.7 
Ohio 6,142 7321 1,654 20.2 385 4.7 
Rhode Island 1,395 71.4 494 25.3 64 3.3 
Tennessee 4,848 76.2 1,210 19.0 303 4.8 
All States 47,511 72.7 14,796 22.7 3,010 4.6 


Note. Percentages are based on the number of valid responses. Missing = 256. 


Table 21. Student Survey ELA PBT (EOY): Q10. Was it easy to flip back and forth between pages to find the story 
or passage that belonged to a question? 


Yes No Did Not Flip Between Pages 
State n % n % n % 
Arizona 5,574 67.7 2,151 26.1 504 6.1 
Arkansas 2,942 69.6 943 22.3 341 8.1 
Colorado 1,237 65.8 507 27.0 136 72 
District of Columbia 170 63.9 80 30.1 16 6.0 
Illinois 5,435 73.0 1,551 20.8 459 6.2 
Louisiana 278 73.9 74 19.7 24 6.4 
Maryland 4,857 67.3 1,992 27.6 365 5.1 
Massachusetts 3,838 70.1 1,364 24.9 274 5.0 
Mississippi 6,178 778 1,406 17.7 355 4.5 
New York 736 69.9 239 22.7 78 74 
Ohio 4,663 72.4 1,362 21.1 418 6.5 
Rhode Island 720 74.6 211 21.9 34 3.5 
Tennessee 1,097 70.8 342 22.1 110 7A 
All States 37,725 71.1 12,222 23.0 3,114 5.9 


Note. Percentages are based on the number of valid responses. Missing = 218. 
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